1 Changelog
- 202408: Collapsing the various ML documents back into one in the hopes that it eases the degree of confusion. All of the exploratory analyses are at the top, at the bottom are the blocks which are used to generate the final tables, but they are only runnable manually due to time constraints.
- 202408: Re-enabled glm, it seems to be running faster again?
- 20240620: Disabled glm training/testing in the container. For reasons I have not yet figured out it takes ~ 10x longer (~10 days) in the container (note, this is a new thing, it used to take ~ 8 hours in the container without error, so something changed in 202405-202406 caret/glmnet/singularity/etc) to complete; which is a problem because my notebooks are auto-regenerated on a 2x/week basis (if changes occur) and so it doesn’t finish before another round starts…
- 202309: Adding a version of this with only the Tumaco data. I am a pretty big non-fan of doing this; so I am going to do so in the most cursory fashion possible by just copying this document to another copy with a suffix of ‘tumaco’ and changing the initial datastructure to the Tumaco-only dataset.
2 Introduction
I had some success in classifying the TMRC2 samples by strain via ML and want to try something more difficult. Thus I will use the normalized gene expression data and try classifying it by cure/fail.
All analyses in this document heavily use caret (Kuhn (2008)) and, as per the sister TMRC2 document, follow pretty closely the path from:
https://topepo.github.io/caret/data-splitting.html#simple-splitting-based-on-the-outcome
and
https://github.com/compgenomr/book/blob/master/05-supervisedLearning.Rmd
3 Starter data
In the strain classifier I used normalized variants. I am thinking to use normalized expression here and therefore explicitly limit myself to ~ 20k variables (significantly down from the 1.6M).
In addition, caret expects the data as (rows == samples) and (columns == variables) where each element is one observation. Thus we will need to transpose the expression matrix.
input_data <- subset_expt(tc_clinical, subset="batch!='biopsy'") %>%
normalize_expt(transform = "log2", convert = "cpm")## The samples excluded are: TMRC30156, TMRC30269, TMRC30253, TMRC30270, TMRC30016, TMRC30017, TMRC30018, TMRC30019, TMRC30020, TMRC30022, TMRC30026, TMRC30044, TMRC30045, TMRC30152, TMRC30177, TMRC30155, TMRC30154, TMRC30241.## subset_expt(): There were 184, now there are 166 samples.## transform_counts: Found 932911 values equal to 0, adding 1 to the matrix.ref_col <- "finaloutcome"
outcome_factor <- as.factor(as.character(pData(input_data)[[ref_col]]))
comparison_n <- 200
## Setting this up in case Najib and Maria Adelaida request a Tumaco set of exploratory analyses.
t_input_data <- subset_expt(tc_clinical, subset="batch!='biopsy'") %>%
normalize_expt(transform = "log2", convert = "cpm")## The samples excluded are: TMRC30156, TMRC30269, TMRC30253, TMRC30270, TMRC30016, TMRC30017, TMRC30018, TMRC30019, TMRC30020, TMRC30022, TMRC30026, TMRC30044, TMRC30045, TMRC30152, TMRC30177, TMRC30155, TMRC30154, TMRC30241.## subset_expt(): There were 184, now there are 166 samples.## transform_counts: Found 932911 values equal to 0, adding 1 to the matrix.t_outcome_factor <- as.factor(as.character(pData(t_input_data)[[ref_col]]))4 Compare TopN important genes to DE enes
Given that we have separated the various analyses, it will take me a minute to figure out where I saved the relevant differential expression analysis. I do not actually save the various DE results to rda files by default, instead opting to send them to xlsx files to share. Recall if you will, that the data that I think might be used for the paper also does not go into the default excel directory but instead mirrors the box organization scheme.
Thus, I think the most relevant file is: “analyses/3_cali_and_tumaco/DE_Cure_vs_Fail/All_Samples/tc_cf_clinical_table_sva-v202207.xlsx”
and
“analyses/4_tumaco/DE_Cure_vs_Fail/All_Samples/t_cf_clinical_table_sva-v202207.xlsx”
Note, this xlsx file contains three primary worksheets, v1cf, v2cf, v3cf which comprise all samples across cell types for each visit and compares cure and fail. When I choose sheet #2 in the following block I am explicitly asking for only the v1 cure/fail result.
4.1 Opening the relevant DE table
input_xlsx <- glue("analyses/3_cali_and_tumaco/DE_Cure_Fail/Clinical_Samples/tc_clinical_cf_sig_sva-v{ver}.xlsx")
all_de_cf <- openxlsx::readWorkbook(input_xlsx, sheet = 3, startRow = 2)
rownames(all_de_cf) <- all_de_cf[["row.names"]]
all_de_cf[["row.names"]] <- NULL
deseq_de_cf_up <- all_de_cf[, c("deseq_logfc", "deseq_adjp", "deseq_basemean", "deseq_lfcse")]
all_de_cf <- openxlsx::readWorkbook(input_xlsx, sheet = 4, startRow = 2)
rownames(all_de_cf) <- all_de_cf[["row.names"]]
all_de_cf[["row.names"]] <- NULL
deseq_de_cf_down <- all_de_cf[, c("deseq_logfc", "deseq_adjp", "deseq_basemean", "deseq_lfcse")]
deseq_de_cf <- rbind.data.frame(deseq_de_cf_up, deseq_de_cf_down)
tumaco_xlsx <- glue("analyses/4_tumaco/DE_Cure_Fail/t_all_visitcf_sig_sva-v{ver}.xlsx")
t_de_cf <- openxlsx::readWorkbook(tumaco_xlsx, sheet = 3, startRow = 2)
rownames(t_de_cf) <- t_de_cf[["row.names"]]
t_de_cf[["row.names"]] <- NULL
t_de_cf_up <- t_de_cf[, c("deseq_logfc", "deseq_adjp", "deseq_basemean", "deseq_lfcse")]
t_de_cf <- openxlsx::readWorkbook(tumaco_xlsx, sheet = 4, startRow = 2)
rownames(t_de_cf) <- t_de_cf[["row.names"]]
t_de_cf[["row.names"]] <- NULL
t_de_cf_down <- t_de_cf[, c("deseq_logfc", "deseq_adjp", "deseq_basemean", "deseq_lfcse")]
t_de_cf <- rbind.data.frame(t_de_cf_up, t_de_cf_down)
top_bottom_ids <- rownames(t_de_cf)4.2 A function to help compare methods to DESeq2
The following block contains a few lists which will hold comparisons between DESeq2 and the various methods which follow, some parameters, and a function to compare the methods and DESeq.
Presumably DESeq and the models should be responding to variance in the data, for which I think the logFC values, p-values, mean values, or standard errors are the most likely proxies to which I have easy access. So, let us pull the top/bottom n genes vis a vis each of those categories and see what happens?
Here is a little function to compare a ML result to the above DE table. It should provide the importance plot, a venn of the two methods, and the gene sets (with annotations) for the unique/shared genes between this ML method and the DE table. Note, this is just using the highest/lowest logFC values and ignoring the p-value for now.
comparison_lfc <- list()
comparison_adjp <- list()
comparison_wgcna <- list()
annotations <- fData(t_monocytes) ## Arbitrarily taken
compare_ml_de <- function(importance, sig_table, annotations, n = comparison_n, plot_n = 15,
according_to = "deseq_logfc", column = "Overall") {
ml_df <- importance[["importance"]] %>%
arrange(desc(!!sym(column)))
importance_plot <- plot(importance, top = plot_n)
top_ml <- head(rownames(ml_df), n = n)
de_df <- sig_table %>%
arrange(desc(deseq_logfc))
top_de <- head(rownames(de_df), n = n)
bottom_de <- tail(rownames(de_df), n = n)
top_bottom_ids <- c(top_de, bottom_de)
comparison <- list("de" = top_bottom_ids, "ml" = top_ml)
comparison_venn <- Vennerable::Venn(comparison)
shared_ids <- comparison_venn@IntersectionSets["11"][[1]]
shared_annot <- annotations[shared_ids, ]
de_ids <- comparison_venn@IntersectionSets["10"][[1]]
ml_ids <- comparison_venn@IntersectionSets["01"][[1]]
only_de <- annotations[de_ids, ]
only_ml <- annotations[ml_ids, ]
comparison_plot <- Vennerable::plot(comparison_venn, doWeights = FALSE, type = "circles")
retlist <- list(
"importance_plot" = importance_plot,
"comparison" = comparison,
"comparison_venn" = comparison_venn,
"comparison_plot" = comparison_plot,
"shared" = shared_annot,
"only_de" = only_de,
"only_ml" = only_ml)
return(retlist)
}5 Filtering
The ML text I am reading provide some neat examples for how one might filter the data to make it more suitable for model creation.
5.1 Near zero variance, or genefilter’s cv
The first filter I was introduced to is quite familiar from our sequencing data, the removal of features with near-zero-variance. Indeed, I am pretty certain that normalize_expt() could do this equivalently and significantly faster than caret::preProcess().
nrow(exprs(input_data))## [1] 19952system.time({
equivalent <- normalize_expt(input_data, filter = "cv", cv_min = 0.1)
})## Removing 3619 low-count genes (16333 remaining).## user system elapsed
## 0.674 0.100 0.773dim(exprs(equivalent))## [1] 16333 166## Given this large amount of data, this step is slow, taking > 10 minutes.
## Yeah seriously, the following three lines get 16,723 genes in 10 minutes while
## the normalize_expt() call above gets 16,749 genes in 2.4 seconds.
#system.time({
# nzv <- preProcess(texprs, method="nzv", uniqueCut=15)
# nzv_texprs <- predict(nzv, texprs)
# dim(nzv_texprs)
#}
nzv_texprs <- t(exprs(equivalent))5.2 Filtering to the highest standard deviation variables
I think I am a bit confused by this filter, one would think that the nzv filter above, if applied correctly, should give you exactly this.
For the moment, I am excluding the following block in order to see how much time/memory keeping these variables costs. If I recall properly, the model of the top-2k variant positions cost ~ 1-4G of memory. I hope that this scales linearly, but I am thinking it might not.
standard_devs <- apply(nzv_texprs, 2, sd)
top_predictors <- order(standard_devs, decreasing = TRUE)[1:3000]
nzv_texprs <- nzv_texprs[, top_predictors]5.3 Center the data
I think centering may not be needed for this data, but here is how:
nzv_center <- preProcess(nzv_texprs, method = "center")
nzv_texprs <- predict(nzv_center, nzv_texprs)6 Merge the appropriate metadata
There are a few metadata factors which might prove of interest for classification. The most obvious are of course outcome, clinic, donor, visit, celltype. I am, for the moment, only likely to focus on outcome. AFAICT I can only include one of these at a time in the data, which is a shame.
interesting_meta <- pData(input_data)[, c("finaloutcome", "donor", "persistence",
"visitnumber", "selectionmethod",
"typeofcells", "time")]
ml_df <- as.data.frame(cbind(outcome_factor, as.data.frame(nzv_uncorr)))
ml_df[["outcome_factor"]] <- as.factor(ml_df[["outcome_factor"]])
dim(ml_df)## [1] 166 9657 Split the data into training/testing
caret provides nice functionality for splitting up the data. I suspect there are many more fun knobs I can play with for instances where I need to exclude some levels of a factor and such. In this case I just want to split by outcome.
7.1 Via data splitting
ml_df <- as.data.frame(cbind(outcome_factor, as.data.frame(nzv_uncorr)))
datasets <- create_partitions(nzv_uncorr, interesting_meta,
outcome_factor = outcome_factor)7.2 Via sampling
There are a few likely sampling methods: cross-validation, bootstrapping, and jackknifing. I will try those out later.
8 Try out training and prediction methods
My goals from here on will be to get the beginnings of a sense of the various methods I can use to create the models from the training data and predict the outcome on the test data. I am hoping also to pick up some idea of what the various arguments mean while I am at it.
8.1 Try out KNN
k-nearest neighbors is somewhat similar to a kmeans estimate. Thus the primary argument is ‘k’
8.1.1 Model creation and performance
split <- 1
train_all <- datasets[["trainers"]][[split]]
train_df <- datasets[["trainers_stripped"]][[split]]
train_idx <- datasets[["train_idx"]][[split]]
train_outcomes <- datasets[["trainer_outcomes"]][[split]]
test_df <- datasets[["testers"]][[split]]
test_idx <- datasets[["test_idx"]][[split]]
test_outcomes <- datasets[["tester_outcomes"]][[split]]
knn_fit <- knn3(x = train_df,
y = train_outcomes,
k = 3)
knn_predict_trained <- predict(knn_fit, train_df, type = "prob")
knn_train_evaluated <- self_evaluate_model(knn_predict_trained, datasets,
which = split, type = "train")## Setting levels: control = cure, case = failure## Setting direction: controls > cases
knn_train_evaluated## The summary of the (in)correct calls is:## Mode FALSE TRUE
## logical 7 60## The missed samples are:## [1] "TMRC30224" "TMRC30071" "TMRC30121" "TMRC30183" "TMRC30181" "TMRC30175"
## [7] "TMRC30144"## The confusion matrix is:## Confusion Matrix and Statistics
##
## Reference
## Prediction cure failure
## cure 40 4
## failure 3 20
##
## Accuracy : 0.896
## 95% CI : (0.797, 0.957)
## No Information Rate : 0.642
## P-Value [Acc > NIR] : 2.28e-06
##
## Kappa : 0.771
##
## Mcnemar's Test P-Value : 1
##
## Sensitivity : 0.930
## Specificity : 0.833
## Pos Pred Value : 0.909
## Neg Pred Value : 0.870
## Precision : 0.909
## Recall : 0.930
## F1 : 0.920
## Prevalence : 0.642
## Detection Rate : 0.597
## Detection Prevalence : 0.657
## Balanced Accuracy : 0.882
##
## 'Positive' Class : cure
## ## The ROC AUC is: 0.949110671936759.As the confusion matrix shows, this failed for a few samples. Perhaps let us change k and see if it improves.
Here is a table of fase positives/negatives for a few values of ‘k’, in this context a false positive is calling a known cure as a failure and false negative is calling a known failure as a cure.
|—|—|—| |k |fp |fn | |2 |0 |8 | |3 |5 |5 | |4 |8 |9 | |5 |11 |7 | |6 |15 |8 |
Note: this depends on the luck of rand(), so the above numbers shift moderately from one run to the next. Thus I think I will just use 2 or 3.
knn_fit2 <- knn3(x = train_df,
y = train_outcomes,
k = 5)
knn_predict_trained2 <- predict(knn_fit2, train_df, type = "prob")
knn_train_evaluated2 <- self_evaluate_model(knn_predict_trained2, datasets,
which = split, type = "train")## Setting levels: control = cure, case = failure## Setting direction: controls > cases
knn_train_evaluated2## The summary of the (in)correct calls is:## Mode FALSE TRUE
## logical 12 55## The missed samples are:## [1] "TMRC30178" "TMRC30223" "TMRC30224" "TMRC30071" "TMRC30080" "TMRC30121"
## [7] "TMRC30048" "TMRC30183" "TMRC30181" "TMRC30218" "TMRC30220" "TMRC30144"## The confusion matrix is:## Confusion Matrix and Statistics
##
## Reference
## Prediction cure failure
## cure 40 4
## failure 8 15
##
## Accuracy : 0.821
## 95% CI : (0.708, 0.904)
## No Information Rate : 0.716
## P-Value [Acc > NIR] : 0.0348
##
## Kappa : 0.586
##
## Mcnemar's Test P-Value : 0.3865
##
## Sensitivity : 0.833
## Specificity : 0.789
## Pos Pred Value : 0.909
## Neg Pred Value : 0.652
## Precision : 0.909
## Recall : 0.833
## F1 : 0.870
## Prevalence : 0.716
## Detection Rate : 0.597
## Detection Prevalence : 0.657
## Balanced Accuracy : 0.811
##
## 'Positive' Class : cure
## ## The ROC AUC is: 0.87302371541502.8.1.2 Predict the rest of the data with this model.
knn_predict_test <- predict(knn_fit, test_df)
knn_test_evaluated <- self_evaluate_model(knn_predict_test, datasets,
which = split, type = "test")## Setting levels: control = cure, case = failure## Setting direction: controls > cases
knn_test_evaluated## The summary of the (in)correct calls is:## Mode FALSE TRUE
## logical 23 76## The missed samples are:## [1] "TMRC30179" "TMRC30222" "TMRC30274" "TMRC30056" "TMRC30105" "TMRC30119"
## [7] "TMRC30103" "TMRC30169" "TMRC30096" "TMRC30118" "TMRC30030" "TMRC30195"
## [13] "TMRC30070" "TMRC30191" "TMRC30132" "TMRC30133" "TMRC30143" "TMRC30146"
## [19] "TMRC30198" "TMRC30237" "TMRC30238" "TMRC30264" "TMRC30147"## The confusion matrix is:## Confusion Matrix and Statistics
##
## Reference
## Prediction cure failure
## cure 54 11
## failure 12 22
##
## Accuracy : 0.768
## 95% CI : (0.672, 0.847)
## No Information Rate : 0.667
## P-Value [Acc > NIR] : 0.0192
##
## Kappa : 0.481
##
## Mcnemar's Test P-Value : 1.0000
##
## Sensitivity : 0.818
## Specificity : 0.667
## Pos Pred Value : 0.831
## Neg Pred Value : 0.647
## Precision : 0.831
## Recall : 0.818
## F1 : 0.824
## Prevalence : 0.667
## Detection Rate : 0.545
## Detection Prevalence : 0.657
## Balanced Accuracy : 0.742
##
## 'Positive' Class : cure
## ## The ROC AUC is: 0.795022624434389.knn_predict_test2 <- predict(knn_fit2, test_df)
knn_test_evaluated2 <- self_evaluate_model(knn_predict_test2, datasets,
which = split, type = "test")## Setting levels: control = cure, case = failure
## Setting direction: controls > cases
knn_test_evaluated2## The summary of the (in)correct calls is:## Mode FALSE TRUE
## logical 36 63## The missed samples are:## [1] "TMRC30179" "TMRC30221" "TMRC30222" "TMRC30056" "TMRC30105" "TMRC30094"
## [7] "TMRC30119" "TMRC30082" "TMRC30103" "TMRC30169" "TMRC30096" "TMRC30083"
## [13] "TMRC30118" "TMRC30014" "TMRC30030" "TMRC30037" "TMRC30054" "TMRC30070"
## [19] "TMRC30041" "TMRC30068" "TMRC30132" "TMRC30123" "TMRC30161" "TMRC30143"
## [25] "TMRC30199" "TMRC30198" "TMRC30201" "TMRC30200" "TMRC30202" "TMRC30204"
## [31] "TMRC30237" "TMRC30238" "TMRC30208" "TMRC30264" "TMRC30147" "TMRC30265"## The confusion matrix is:## Confusion Matrix and Statistics
##
## Reference
## Prediction cure failure
## cure 54 11
## failure 25 9
##
## Accuracy : 0.636
## 95% CI : (0.534, 0.731)
## No Information Rate : 0.798
## P-Value [Acc > NIR] : 0.9999
##
## Kappa : 0.106
##
## Mcnemar's Test P-Value : 0.0303
##
## Sensitivity : 0.684
## Specificity : 0.450
## Pos Pred Value : 0.831
## Neg Pred Value : 0.265
## Precision : 0.831
## Recall : 0.684
## F1 : 0.750
## Prevalence : 0.798
## Detection Rate : 0.545
## Detection Prevalence : 0.657
## Balanced Accuracy : 0.567
##
## 'Positive' Class : cure
## ## The ROC AUC is: 0.69683257918552.8.2 Perform cross-validation to estimate k
The cross validation method of repeated sampling the data is all done within the train() function. With that in mind, here it is operating with the knn method.
8.2.1 CV with knn
When train() is called with the trControl and tuneGrid, we can control how the knn training is repeated, in this case it will iterate over k from 1 to 10.
This currently fails due to a stack overflow…
cv_control <- trainControl(method = "cv", number = 10)
knn_train_fit <- train(outcome_factor ~ ., data = train_df,
method = "knn",
trControl = cv_control,
tuneGrid = data.frame(k = 1:10))
knn_train_fit[["bestTune"]]
plot(x = 1:10, 1 - knn_train_fit$results[, 2], pch = 19,
ylab = "prediction error", xlab = "k")
lines(loess.smooth(x = 1:10, 1 - knn_train_fit$results[, 2],degree = 2),
col = "#CC0000")8.2.2 Bootstrap with knn
boot_control <- trainControl(method = "boot", number = 20,
returnResamp = "all")
knn_train_fit <- train(outcome ~ ., data = train_all,
method = "knn",
trControl = boot_control,
tuneGrid = data.frame(k = 1:10))
knn_train_fit[["bestTune"]]## k
## 1 1plot(x = 1:10, 1 - knn_train_fit$results[, 2], pch = 19,
ylab = "prediction error", xlab = "k")
lines(loess.smooth(x = 1:10, 1 - knn_train_fit$results[, 2],degree = 2),
col = "#CC0000")
8.2.3 Explain the important variables
In this instance we will search for genes which were important for the model’s creation.
The DALEX package provides a function: feature_importance() which seeks to use a series of other methods to extract (in this case, genes) features which do a good job of explaining the result produced by the model. In the case of this dataset, which has thousands of features, this does not appear to end well.
explainer_knn <- DALEX::explain(knn_fit, label = "knn",
data = train_df,
y = as.numeric(train_outcomes))
## AFAICT the following will take forever unless we drastically reduce the complexity of the model.
## yeah, I let it run for a week.
## features <- feature_importance(explainer_knn, n_sample = 50, type = "difference")Conversely, we can use the varImp() function…
knn_variable_importance <- varImp(knn_train_fit)
plot(knn_variable_importance, top = 15)
knn_variables <- knn_variable_importance[["importance"]] %>%
arrange(desc(cure))Given that, we can ask for the similarity to the DESeq results…
knn_comparison <- compare_ml_de(knn_variable_importance, deseq_de_cf, annotations,
n = comparison_n, plot_n = 15, column = "cure")
knn_comparison[["comparison_plot"]]## Vennmarcomparison_lfc[["knn"]] <- knn_comparison[["shared"]]8.3 Random Forest
I think R/caret provides a few implementation of the forest family of methods, this is using ranger(Wright and Ziegler (2017)).
The parameter ‘mtry’ is often important, if I read the text correctly it controls how many variables to sample in each split of the tree. Thus higher numbers should presumably make it more specific at the risk of overfitting.
Setting min.node.size sets the minimume node size of terminal nodes in each tree. Each increment up speeds the algorithm.
I am going to use my boot control trainer from above and see how it goes.
rf_train_fit <- train(outcome ~ ., data = train_all,
method = "ranger", trControl = boot_control,
importance = "permutation",
tuneGrid = data.frame(
mtry = 200,
min.node.size = 1,
splitrule = "gini"),
verbose = TRUE)
rf_train_fit[["finalModel"]][["prediction.error"]]## [1] 0.2388rf_variable_importance <- varImp(rf_train_fit)
plot(rf_variable_importance, top = 15)
rf_variables <- rf_variable_importance[["importance"]] %>%
arrange(desc(Overall))
rf_predict_trained <- predict(rf_train_fit, train_df)
rf_predict_evaluated <- self_evaluate_model(rf_predict_trained, datasets,
which = split, type = "train")## Setting levels: control = cure, case = failure## Setting direction: controls < cases
rf_predict_evaluated## The summary of the (in)correct calls is:## Mode TRUE
## logical 67## The missed samples are:## character(0)## The confusion matrix is:## Confusion Matrix and Statistics
##
## Reference
## Prediction cure failure
## cure 44 0
## failure 0 23
##
## Accuracy : 1
## 95% CI : (0.946, 1)
## No Information Rate : 0.657
## P-Value [Acc > NIR] : 5.81e-13
##
## Kappa : 1
##
## Mcnemar's Test P-Value : NA
##
## Sensitivity : 1.000
## Specificity : 1.000
## Pos Pred Value : 1.000
## Neg Pred Value : 1.000
## Precision : 1.000
## Recall : 1.000
## F1 : 1.000
## Prevalence : 0.657
## Detection Rate : 0.657
## Detection Prevalence : 0.657
## Balanced Accuracy : 1.000
##
## 'Positive' Class : cure
## ## The ROC AUC is: 1.8.3.1 What would be shared between DESeq2 and the ML classifier?
rf_comparison <- compare_ml_de(rf_variable_importance, deseq_de_cf, annotations,
n = comparison_n, plot_n = 15)
rf_comparison[["comparison_venn"]]## A Venn object on 2 sets named
## de,ml
## 00 10 01 11
## 0 249 167 33comparison_lfc[["rf"]] <- rf_comparison[["shared"]]8.3.2 Compare to the WGCNA results
A couple months ago I spent a little time attempting to recapitulate Alejandro’s WGCNA results. I think I did so by mostly copy/pasting his work and adding some commentary and tweaking parts of it so that it was easier for me to read/understand. In the process, I generated a series of modules which looked similar/identical to his. Unfortunately, I did not add some sections to record the genes/modules to some output files. I am therefore going back to that now and doing so in the hopes that I can compare those modules to the results produced by the clasifiers.
It seems that the interest in this comparison has waned, so I am just going to disable it.
wgcna_result <- openxlsx::readWorkbook(glue("excel/wgcna_interesting_genes-v{ver}.xlsx"))
rownames(wgcna_result) <- wgcna_result[["row.names"]]
wgcna_result[["row.names"]] <- NULL
top_ml <- rownames(head(rf_variables, n = comparison_n))
comparison <- list("wgcna" = rownames(wgcna_result), "ml" = top_ml)
comparison_venn <- Vennerable::Venn(comparison)
comparison_wgcna[["rf"]] <- annotations[comparison_venn@IntersectionSets["11"][[1]], ]
Vennerable::plot(comparison_venn, doWeights = FALSE, type = "circles")8.3.2.1 Digression do the genes provided by varImp mean anything?
Let us take a moment and see if the top-n genes returned by varImp() have some meaning which jumps out. One might assume, given our extant Differential Expression results, that the interleukin response will be a likely candidate.
importance_gp <- simple_gprofiler(rownames(head(rf_variables, n = comparison_n)))
importance_gp## A set of ontologies produced by gprofiler using 200
## genes against the hsapiens annotations and significance cutoff 0.05.
## There are:
## 5 MF
## 49 BP
## 2 KEGG
## 4 REAC
## 1 WP
## 9 TF
## 0 MIRNA
## 0 HPA
## 2 CORUM
## 0 HP hits.8.3.3 Now the random forest testers!
rf_predict_test <- predict(rf_train_fit, test_df)
rf_predict_test_evaluated <- self_evaluate_model(rf_predict_test, datasets,
which = split, type = "test")## Setting levels: control = cure, case = failure## Setting direction: controls < cases
rf_predict_test_evaluated## The summary of the (in)correct calls is:## Mode FALSE TRUE
## logical 14 85## The missed samples are:## [1] "TMRC30179" "TMRC30222" "TMRC30105" "TMRC30058" "TMRC30094" "TMRC30119"
## [7] "TMRC30103" "TMRC30068" "TMRC30198" "TMRC30200" "TMRC30202" "TMRC30204"
## [13] "TMRC30238" "TMRC30265"## The confusion matrix is:## Confusion Matrix and Statistics
##
## Reference
## Prediction cure failure
## cure 64 1
## failure 13 21
##
## Accuracy : 0.859
## 95% CI : (0.774, 0.92)
## No Information Rate : 0.778
## P-Value [Acc > NIR] : 0.03028
##
## Kappa : 0.658
##
## Mcnemar's Test P-Value : 0.00328
##
## Sensitivity : 0.831
## Specificity : 0.955
## Pos Pred Value : 0.985
## Neg Pred Value : 0.618
## Precision : 0.985
## Recall : 0.831
## F1 : 0.901
## Prevalence : 0.778
## Detection Rate : 0.646
## Detection Prevalence : 0.657
## Balanced Accuracy : 0.893
##
## 'Positive' Class : cure
## ## The ROC AUC is: 0.801131221719457.8.4 GLM, or Logistic regression and regularization
Logistic regression is a statistical method for binary responses. However, it is able to work with multiple classes as well. The general idea of this method is to find parameters which increase the likelihood that the observed data is sampled from a statistical distribution of interest. The transformations and linear regression-esque tasks performed are confusing, but once those are performed, the task becomes setting the model’s (fitting) parameters to values which increase the probability that the statistical model looks like the actual dataset given the training data, and that when samples, will return values which are similar. The most likely statistical distributions one will want to fit are the Gaussian, in which case we want to transform/normalize the mean/variance of our variables so they look whatever normal distribution we are using. Conversely, logistic regression uses a binnomial distribution (like our raw sequencing data!) but which is from 0-1.
8.4.1 Using a single gene
Let us take the most important gene observed in one of our previous training sets: ENSG00000248405 PRR5-ARHGAP8
gene_id <- "ENSG00000248405"
single_fit <- train(
outcome ~ ENSG00000248405, data = train_all,
method = "glm", family = "binomial", trControl = trainControl("none"))
tt <- data.frame("ENSG00000248405" = seq(min(train_df[[gene_id]]),
max(train_df[[gene_id]]), len = 100))
## predict probabilities for the simulated data
tt$subtype = predict(single_fit, newdata = tt, type="prob")[, 1]
## plot the sigmoid curve and the training data
plot(ifelse(outcome == "cure", 1, 0) ~ ENSG00000248405,
data = train_all, col = "red4",
ylab = "CF as 0 or 1", xlab = "favorite gene expression")
lines(subtype ~ ENSG00000248405, tt, col = "green4", lwd = 2)
plot_df <- train_all[, c("outcome", "ENSG00000248405")]
ggbetweenstats(plot_df, "outcome", "ENSG00000248405")Having tried with 1 gene, let us extend this to all genes. In my first try of this, it took a long time.
glm_train_fit <- train(outcome ~ ., data = train_all,
trControl = boot_control,
method = "glm", family = "binomial")## Warning in predict.lm(object, newdata, se.fit, scale = 1, type = if (type == :
## prediction from rank-deficient fit; attr(*, "non-estim") has doubtful cases
## Warning in predict.lm(object, newdata, se.fit, scale = 1, type = if (type == :
## prediction from rank-deficient fit; attr(*, "non-estim") has doubtful cases
## Warning in predict.lm(object, newdata, se.fit, scale = 1, type = if (type == :
## prediction from rank-deficient fit; attr(*, "non-estim") has doubtful cases
## Warning in predict.lm(object, newdata, se.fit, scale = 1, type = if (type == :
## prediction from rank-deficient fit; attr(*, "non-estim") has doubtful cases
## Warning in predict.lm(object, newdata, se.fit, scale = 1, type = if (type == :
## prediction from rank-deficient fit; attr(*, "non-estim") has doubtful cases
## Warning in predict.lm(object, newdata, se.fit, scale = 1, type = if (type == :
## prediction from rank-deficient fit; attr(*, "non-estim") has doubtful cases
## Warning in predict.lm(object, newdata, se.fit, scale = 1, type = if (type == :
## prediction from rank-deficient fit; attr(*, "non-estim") has doubtful cases
## Warning in predict.lm(object, newdata, se.fit, scale = 1, type = if (type == :
## prediction from rank-deficient fit; attr(*, "non-estim") has doubtful cases
## Warning in predict.lm(object, newdata, se.fit, scale = 1, type = if (type == :
## prediction from rank-deficient fit; attr(*, "non-estim") has doubtful cases
## Warning in predict.lm(object, newdata, se.fit, scale = 1, type = if (type == :
## prediction from rank-deficient fit; attr(*, "non-estim") has doubtful cases
## Warning in predict.lm(object, newdata, se.fit, scale = 1, type = if (type == :
## prediction from rank-deficient fit; attr(*, "non-estim") has doubtful cases
## Warning in predict.lm(object, newdata, se.fit, scale = 1, type = if (type == :
## prediction from rank-deficient fit; attr(*, "non-estim") has doubtful cases
## Warning in predict.lm(object, newdata, se.fit, scale = 1, type = if (type == :
## prediction from rank-deficient fit; attr(*, "non-estim") has doubtful cases
## Warning in predict.lm(object, newdata, se.fit, scale = 1, type = if (type == :
## prediction from rank-deficient fit; attr(*, "non-estim") has doubtful cases
## Warning in predict.lm(object, newdata, se.fit, scale = 1, type = if (type == :
## prediction from rank-deficient fit; attr(*, "non-estim") has doubtful cases
## Warning in predict.lm(object, newdata, se.fit, scale = 1, type = if (type == :
## prediction from rank-deficient fit; attr(*, "non-estim") has doubtful cases
## Warning in predict.lm(object, newdata, se.fit, scale = 1, type = if (type == :
## prediction from rank-deficient fit; attr(*, "non-estim") has doubtful cases
## Warning in predict.lm(object, newdata, se.fit, scale = 1, type = if (type == :
## prediction from rank-deficient fit; attr(*, "non-estim") has doubtful cases
## Warning in predict.lm(object, newdata, se.fit, scale = 1, type = if (type == :
## prediction from rank-deficient fit; attr(*, "non-estim") has doubtful cases
## Warning in predict.lm(object, newdata, se.fit, scale = 1, type = if (type == :
## prediction from rank-deficient fit; attr(*, "non-estim") has doubtful cases8.5 Compare GLM and WGCNA/DE
The previous block does not take into account our parameter sweep, so it isn’t theoretically very useful for this series of methological tests. As a result, the following block is also not particularly helpful – though I suspect the results are identical to what follows.
glm_variable_importance <- varImp(glm_train_fit)
## Oh, this only produces 100 entries -- so me getting the top 400 is silly.
glm_variables <- glm_variable_importance[["importance"]] %>%
arrange(desc(Overall))
tt = plot(glm_variable_importance, top = 15)
simple_gprofiler(rownames(head(glm_variables, n = comparison_n)))## A set of ontologies produced by gprofiler using 66
## genes against the hsapiens annotations and significance cutoff 0.05.
## There are:
## 2 MF
## 123 BP
## 1 KEGG
## 3 REAC
## 5 WP
## 1 TF
## 0 MIRNA
## 3 HPA
## 0 CORUM
## 1 HP hits.top_glm <- rownames(glm_variables)
comparison <- list("de" = top_bottom_ids, "ml" = top_glm)
comparison_venn <- Vennerable::Venn(comparison)
comparison_lfc[["glm"]] <- fData(c_monocytes)[comparison_venn@IntersectionSets["11"][[1]], ]
Vennerable::plot(comparison_venn, doWeights = FALSE, type = "circles")
Dropping the adjp and wgcna comparisons, they show little overlap.
comparison <- list("de" = lowest_adjp, "ml" = top_glm)
comparison_venn <- Vennerable::Venn(comparison)
comparison_adjp[["glm"]] <- fData(c_monocytes)[comparison_venn@IntersectionSets["11"][[1]], ]
Vennerable::plot(comparison_venn, doWeights = FALSE, type = "circles")
comparison <- list("de" = highest_exprs, "ml" = top_glm)
comparison_venn <- Vennerable::Venn(comparison)
Vennerable::plot(comparison_venn, doWeights = FALSE, type = "circles")In this block, I repeat the above with the tuneGrid option set so that it may do the training using a sweep of putatively helpful parameters; as a result it should provide a more appropriate (but quite possibly identical) trainer.
##rf_method <- trainControl(method = "ranger", number = 10, verbose = TRUE)
## train_method <- trainControl(method = "cv", number = 10)
glm_fit <- train(outcome ~ ., data = train_all, method = "glmnet",
trControl = boot_control, importance = "permutation",
tuneGrid = data.frame(
alpha = 0.5,
lambda = seq(0.1, 0.7, 0.05)),
verbose = TRUE)
glm_fit## glmnet
##
## 67 samples
## 964 predictors
## 2 classes: 'cure', 'failure'
##
## No pre-processing
## Resampling: Bootstrapped (20 reps)
## Summary of sample sizes: 67, 67, 67, 67, 67, 67, ...
## Resampling results across tuning parameters:
##
## lambda Accuracy Kappa
## 0.10 0.7686 0.477806
## 0.15 0.7264 0.367329
## 0.20 0.6833 0.243122
## 0.25 0.6592 0.152945
## 0.30 0.6479 0.087507
## 0.35 0.6314 0.015878
## 0.40 0.6313 0.002206
## 0.45 0.6313 0.000000
## 0.50 0.6313 0.000000
## 0.55 0.6313 0.000000
## 0.60 0.6313 0.000000
## 0.65 0.6313 0.000000
## 0.70 0.6313 0.000000
##
## Tuning parameter 'alpha' was held constant at a value of 0.5
## Accuracy was used to select the optimal model using the largest value.
## The final values used for the model were alpha = 0.5 and lambda = 0.1.glm_predict_trained <- predict(glm_fit, train_df)
glm_train_eval <- self_evaluate_model(glm_predict_trained, datasets,
which = split, type = "train")## Setting levels: control = cure, case = failure## Setting direction: controls < cases
glm_train_eval## The summary of the (in)correct calls is:## Mode FALSE TRUE
## logical 1 66## The missed samples are:## [1] "TMRC30220"## The confusion matrix is:## Confusion Matrix and Statistics
##
## Reference
## Prediction cure failure
## cure 44 0
## failure 1 22
##
## Accuracy : 0.985
## 95% CI : (0.92, 1)
## No Information Rate : 0.672
## P-Value [Acc > NIR] : 8.84e-11
##
## Kappa : 0.967
##
## Mcnemar's Test P-Value : 1
##
## Sensitivity : 0.978
## Specificity : 1.000
## Pos Pred Value : 1.000
## Neg Pred Value : 0.957
## Precision : 1.000
## Recall : 0.978
## F1 : 0.989
## Prevalence : 0.672
## Detection Rate : 0.657
## Detection Prevalence : 0.657
## Balanced Accuracy : 0.989
##
## 'Positive' Class : cure
## ## The ROC AUC is: 0.978260869565217.8.5.1 Now the GLM testers!
glm_predict_test <- predict(glm_fit, test_df)
glm_fit_eval_test <- self_evaluate_model(glm_predict_test, datasets,
which = split, type = "test")## Setting levels: control = cure, case = failure## Setting direction: controls < cases
glm_fit_eval_test## The summary of the (in)correct calls is:## Mode FALSE TRUE
## logical 15 84## The missed samples are:## [1] "TMRC30179" "TMRC30221" "TMRC30222" "TMRC30058" "TMRC30094" "TMRC30143"
## [7] "TMRC30146" "TMRC30199" "TMRC30198" "TMRC30201" "TMRC30200" "TMRC30202"
## [13] "TMRC30238" "TMRC30264" "TMRC30265"## The confusion matrix is:## Confusion Matrix and Statistics
##
## Reference
## Prediction cure failure
## cure 63 2
## failure 13 21
##
## Accuracy : 0.848
## 95% CI : (0.762, 0.913)
## No Information Rate : 0.768
## P-Value [Acc > NIR] : 0.03273
##
## Kappa : 0.636
##
## Mcnemar's Test P-Value : 0.00982
##
## Sensitivity : 0.829
## Specificity : 0.913
## Pos Pred Value : 0.969
## Neg Pred Value : 0.618
## Precision : 0.969
## Recall : 0.829
## F1 : 0.894
## Prevalence : 0.768
## Detection Rate : 0.636
## Detection Prevalence : 0.657
## Balanced Accuracy : 0.871
##
## 'Positive' Class : cure
## ## The ROC AUC is: 0.793438914027149.8.5.2 Compare again vs the DE table
In the following block I extract the top and bottom DE genes vs. the top set of genes in the glm_variables ordered by the ‘Overall’ column.
I should make this a function, there is no way this will kept correct across multiple iterations of the ml/wgcna/etc comparisons.
I am going to stop trying to compare these results across WGCNA/adjp/expression, those comparisons are all basically null.
glm_variable_importance <- varImp(glm_fit)
glm_comparison <- compare_ml_de(glm_variable_importance, deseq_de_cf, annotations,
n = comparison_n, plot_n = 15)
comparison_lfc[["glm"]] <- glm_comparison[["shared"]]8.6 Gradient Booster
At this point I basically have a feeling for how one may use the tuneGrid to apply a parameter sweep of the data along with the various (in this case cv) sampling methods. Thus, the following is basically identical to the previous blocks except it is using xgbTree (which is the R implementation of extreme boosting: (Chen and Guestrin (2016))).
##rf_method <- trainControl(method = "ranger", number = 10, verbose = TRUE)
train_method <- trainControl(method = "cv", number = 10)
gb_fit <- train(outcome ~ ., data = train_all,
method = "xgbTree", trControl = train_method,
tuneGrid = data.frame(
nrounds = 200,
eta = c(0.05, 0.1, 0.3),
max_depth = 4,
gamma = 0,
colsample_bytree = 1,
subsample = 0.5,
min_child_weight = 1),
verbose = TRUE)
gb_predict_trained <- predict(gb_fit, train_df)
gb_predict_trained## [1] cure failure failure failure cure cure cure cure cure
## [10] cure cure cure cure cure cure cure cure cure
## [19] cure cure cure cure cure cure failure cure failure
## [28] cure cure failure failure cure cure cure cure cure
## [37] failure failure failure failure failure failure cure cure cure
## [46] cure cure failure cure failure cure cure failure cure
## [55] cure cure cure failure cure failure failure failure failure
## [64] failure failure cure cure
## Levels: cure failuregb_train_eval <- self_evaluate_model(gb_predict_trained, datasets,
which = split, type = "train")## Setting levels: control = cure, case = failure## Setting direction: controls < cases
gb_train_eval## The summary of the (in)correct calls is:## Mode TRUE
## logical 67## The missed samples are:## character(0)## The confusion matrix is:## Confusion Matrix and Statistics
##
## Reference
## Prediction cure failure
## cure 44 0
## failure 0 23
##
## Accuracy : 1
## 95% CI : (0.946, 1)
## No Information Rate : 0.657
## P-Value [Acc > NIR] : 5.81e-13
##
## Kappa : 1
##
## Mcnemar's Test P-Value : NA
##
## Sensitivity : 1.000
## Specificity : 1.000
## Pos Pred Value : 1.000
## Neg Pred Value : 1.000
## Precision : 1.000
## Recall : 1.000
## F1 : 1.000
## Prevalence : 0.657
## Detection Rate : 0.657
## Detection Prevalence : 0.657
## Balanced Accuracy : 1.000
##
## 'Positive' Class : cure
## ## The ROC AUC is: 1.8.6.1 Now the GB testers!
gb_predict_test <- predict(gb_fit, test_df)
gb_predict_test_evaluated <- self_evaluate_model(gb_predict_test, datasets,
which = split, type = "test")## Setting levels: control = cure, case = failure## Setting direction: controls < cases
gb_predict_test_evaluated## The summary of the (in)correct calls is:## Mode FALSE TRUE
## logical 13 86## The missed samples are:## [1] "TMRC30179" "TMRC30222" "TMRC30105" "TMRC30058" "TMRC30094" "TMRC30119"
## [7] "TMRC30197" "TMRC30201" "TMRC30200" "TMRC30204" "TMRC30238" "TMRC30264"
## [13] "TMRC30265"## The confusion matrix is:## Confusion Matrix and Statistics
##
## Reference
## Prediction cure failure
## cure 65 0
## failure 13 21
##
## Accuracy : 0.869
## 95% CI : (0.786, 0.928)
## No Information Rate : 0.788
## P-Value [Acc > NIR] : 0.027769
##
## Kappa : 0.68
##
## Mcnemar's Test P-Value : 0.000874
##
## Sensitivity : 0.833
## Specificity : 1.000
## Pos Pred Value : 1.000
## Neg Pred Value : 0.618
## Precision : 1.000
## Recall : 0.833
## F1 : 0.909
## Prevalence : 0.788
## Detection Rate : 0.657
## Detection Prevalence : 0.657
## Balanced Accuracy : 0.917
##
## 'Positive' Class : cure
## ## The ROC AUC is: 0.808823529411765.gb_variable_importance <- varImp(gb_fit)
gb_variables <- glm_variable_importance[["importance"]] %>%
arrange(desc(Overall))
plot(gb_variable_importance, top = 15)
gb_comparison <- compare_ml_de(gb_variable_importance, deseq_de_cf, annotations, n = comparison_n, plot_n = 15)
comparison_lfc[["gb"]] <- gb_comparison[["shared"]]9 Shared importance
upset_lfc <- list()
upset_adjp <- list()
upset_wgcna <- list()
for (d in 1:length(comparison_lfc)) {
name <- names(comparison_lfc)[d]
upset_lfc[[name]] <- rownames(comparison_lfc[[name]])
upset_adjp[[name]] <- rownames(comparison_adjp[[name]])
##upset_wgcna[[name]] <- rownames(comparison_wgcna[[name]])
}
start_lfc <- UpSetR::fromList(upset_lfc)
lfc_vs_ml <- UpSetR::upset(start_lfc)9.1 A different way to query shared/unique genes
The various gene sets observed by these methods may also be compared directly by an upset plot using the deseq_de_cf with (gb|glm|rf|knn)_variable_importance…
comp_number <- 300
de_ml_sig_list <- list(
"DESeq2" = rownames(deseq_de_cf),
"knn" = rownames(head(knn_variables, n = comp_number)),
"rf" = rownames(head(rf_variables, n = comp_number)),
"glm" = rownames(head(glm_variables, n = comp_number)),
"gb" = rownames(head(gb_variables, n = comp_number)))
important_genes_upset_input <- UpSetR::fromList(de_ml_sig_list)
important_genes_upset <- UpSetR::upset(important_genes_upset_input)
print(important_genes_upset)
important_genes_overlap <- overlap_groups(de_ml_sig_list)
shared_gene_ids <- overlap_geneids(important_genes_overlap, "DESeq2:knn")
shared_annotations <- fData(input_data)[as.character(shared_gene_ids), ]
important_genes_overlap <- overlap_groups(de_ml_sig_list)
shared_gene_ids <- overlap_geneids(important_genes_overlap, "DESeq2:rf")
shared_annotations <- fData(input_data)[as.character(shared_gene_ids), ]
important_genes_overlap <- overlap_groups(de_ml_sig_list)
shared_gene_ids <- overlap_geneids(important_genes_overlap, "DESeq2:gb")
shared_annotations <- fData(input_data)[as.character(shared_gene_ids), ]
important_genes_overlap <- overlap_groups(de_ml_sig_list)
shared_gene_ids <- overlap_geneids(important_genes_overlap, "DESeq2:glm")
shared_annotations <- fData(input_data)[as.character(shared_gene_ids), ]11 Performing multiple train/test rounds in one call
Doing all of the above for each method and collecting the results is super fiddly, so I wrote a little function to try to make that easier.
WARNING: Turning on the following block takes ~ 2 days on our pretty nice server. Your mileage may vary.
11.1 Setting up input data for multiple ML iterations
There are four versions of the data that Maria Adelaida and Najib are interested in:
- Tumaco+Cali all visits
- Tumaco+Cali visit 1
- Tumaco all visits
- Tumaco visit 1
Ergo, let us set up an appropriate naming convention for them to make these iterative runs more coherent. I will continue using ‘tc’ and ‘t’ for the clinics as a prefix and follow it with ‘vall’ vs. ‘v1’.
tc_vall_input <- subset_expt(tc_clinical, subset = "batch!='biopsy'")## The samples excluded are: TMRC30156, TMRC30269, TMRC30253, TMRC30270, TMRC30016, TMRC30017, TMRC30018, TMRC30019, TMRC30020, TMRC30022, TMRC30026, TMRC30044, TMRC30045, TMRC30152, TMRC30177, TMRC30155, TMRC30154, TMRC30241.## subset_expt(): There were 184, now there are 166 samples.t_vall_input <- subset_expt(tc_vall_input, subset = "clinic=='tumaco'")## The samples excluded are: TMRC30185, TMRC30186, TMRC30178, TMRC30179, TMRC30221, TMRC30222, TMRC30223, TMRC30224, TMRC30148, TMRC30149, TMRC30150, TMRC30140, TMRC30138, TMRC30176, TMRC30153, TMRC30151, TMRC30234, TMRC30235, TMRC30225, TMRC30226, TMRC30227, TMRC30228, TMRC30229, TMRC30230, TMRC30231, TMRC30232, TMRC30233, TMRC30209, TMRC30210, TMRC30211, TMRC30212, TMRC30213, TMRC30216, TMRC30214, TMRC30215, TMRC30271, TMRC30273, TMRC30275, TMRC30272, TMRC30274, TMRC30276, TMRC30254, TMRC30255, TMRC30256, TMRC30277, TMRC30239, TMRC30240, TMRC30278, TMRC30279, TMRC30280, TMRC30257, TMRC30258, TMRC30281, TMRC30283, TMRC30284, TMRC30282, TMRC30285.## subset_expt(): There were 166, now there are 109 samples.tc_v1_input <- subset_expt(tc_vall_input, subset = "visitbipart=='v1'")## The samples excluded are: TMRC30178, TMRC30179, TMRC30223, TMRC30224, TMRC30150, TMRC30140, TMRC30176, TMRC30153, TMRC30151, TMRC30228, TMRC30229, TMRC30230, TMRC30231, TMRC30232, TMRC30233, TMRC30212, TMRC30213, TMRC30216, TMRC30214, TMRC30215, TMRC30272, TMRC30274, TMRC30276, TMRC30254, TMRC30255, TMRC30256, TMRC30277, TMRC30278, TMRC30279, TMRC30280, TMRC30282, TMRC30285, TMRC30056, TMRC30113, TMRC30105, TMRC30058, TMRC30164, TMRC30094, TMRC30119, TMRC30082, TMRC30122, TMRC30169, TMRC30093, TMRC30170, TMRC30032, TMRC30096, TMRC30028, TMRC30115, TMRC30118, TMRC30121, TMRC30196, TMRC30030, TMRC30037, TMRC30031, TMRC30027, TMRC30194, TMRC30195, TMRC30054, TMRC30070, TMRC30049, TMRC30055, TMRC30053, TMRC30068, TMRC30171, TMRC30139, TMRC30158, TMRC30160, TMRC30157, TMRC30183, TMRC30181, TMRC30072, TMRC30133, TMRC30078, TMRC30076, TMRC30159, TMRC30129, TMRC30088, TMRC30172, TMRC30137, TMRC30161, TMRC30142, TMRC30145, TMRC30143, TMRC30146, TMRC30182, TMRC30199, TMRC30201, TMRC30200, TMRC30202, TMRC30205, TMRC30206, TMRC30136, TMRC30217, TMRC30077, TMRC30219, TMRC30218, TMRC30079, TMRC30220, TMRC30173, TMRC30144, TMRC30147.## subset_expt(): There were 166, now there are 65 samples.t_v1_input <- subset_expt(t_vall_input, subset = "visitbipart=='v1'")## The samples excluded are: TMRC30056, TMRC30113, TMRC30105, TMRC30058, TMRC30164, TMRC30094, TMRC30119, TMRC30082, TMRC30122, TMRC30169, TMRC30093, TMRC30170, TMRC30032, TMRC30096, TMRC30028, TMRC30115, TMRC30118, TMRC30121, TMRC30196, TMRC30030, TMRC30037, TMRC30031, TMRC30027, TMRC30194, TMRC30195, TMRC30054, TMRC30070, TMRC30049, TMRC30055, TMRC30053, TMRC30068, TMRC30171, TMRC30139, TMRC30158, TMRC30160, TMRC30157, TMRC30183, TMRC30181, TMRC30072, TMRC30133, TMRC30078, TMRC30076, TMRC30159, TMRC30129, TMRC30088, TMRC30172, TMRC30137, TMRC30161, TMRC30142, TMRC30145, TMRC30143, TMRC30146, TMRC30182, TMRC30199, TMRC30201, TMRC30200, TMRC30202, TMRC30205, TMRC30206, TMRC30136, TMRC30217, TMRC30077, TMRC30219, TMRC30218, TMRC30079, TMRC30220, TMRC30173, TMRC30144, TMRC30147.## subset_expt(): There were 109, now there are 40 samples.tc_vall_input <- normalize_expt(tc_vall_input, transform = "log2", convert = "cpm")## transform_counts: Found 932911 values equal to 0, adding 1 to the matrix.t_vall_input <- normalize_expt(t_vall_input, transform = "log2", convert = "cpm")## transform_counts: Found 587592 values equal to 0, adding 1 to the matrix.tc_v1_input <- normalize_expt(tc_v1_input, transform = "log2", convert = "cpm")## transform_counts: Found 369543 values equal to 0, adding 1 to the matrix.t_v1_input <- normalize_expt(t_v1_input, transform = "log2", convert = "cpm")## transform_counts: Found 216947 values equal to 0, adding 1 to the matrix.tc_vall_outcome_factor <- as.factor(as.character(pData(tc_vall_input)[[ref_col]]))
t_vall_outcome_factor <- as.factor(as.character(pData(t_vall_input)[[ref_col]]))
tc_v1_outcome_factor <- as.factor(as.character(pData(tc_v1_input)[[ref_col]]))
t_v1_outcome_factor <- as.factor(as.character(pData(t_v1_input)[[ref_col]]))11.2 Perform normalization/filtering operations as per the exploratory document
In the following block I will use the same normalizations and filters as above on each of the four inputs.
tc_vall_norm <- normalize_expt(tc_vall_input, filter = "cv", cv_min = 0.1)## Removing 3619 low-count genes (16333 remaining).tc_v1_norm <- normalize_expt(tc_v1_input, filter = "cv", cv_min = 0.1)## Removing 3968 low-count genes (15984 remaining).t_vall_norm <- normalize_expt(t_vall_input, filter = "cv", cv_min = 0.1)## Removing 3785 low-count genes (16167 remaining).t_v1_norm <- normalize_expt(t_v1_input, filter = "cv", cv_min = 0.1)## Removing 4109 low-count genes (15843 remaining).tc_vall_texprs <- t(exprs(tc_vall_norm))
tc_v1_texprs <- t(exprs(tc_v1_norm))
t_vall_texprs <- t(exprs(t_vall_norm))
t_v1_texprs <- t(exprs(t_v1_norm))
sd_genes <- 3000
tc_vall_stdev <- apply(tc_vall_texprs, 2, sd)
tc_vall_pred <- order(tc_vall_stdev, decreasing = TRUE)[1:sd_genes]
tc_vall_texprs <- tc_vall_texprs[, tc_vall_pred]
tc_v1_stdev <- apply(tc_v1_texprs, 2, sd)
tc_v1_pred <- order(tc_v1_stdev, decreasing = TRUE)[1:sd_genes]
tc_v1_texprs <- tc_v1_texprs[, tc_v1_pred]
t_vall_stdev <- apply(t_vall_texprs, 2, sd)
t_vall_pred <- order(t_vall_stdev, decreasing = TRUE)[1:sd_genes]
t_vall_texprs <- t_vall_texprs[, t_vall_pred]
t_v1_stdev <- apply(t_v1_texprs, 2, sd)
t_v1_pred <- order(t_v1_stdev, decreasing = TRUE)[1:sd_genes]
t_v1_texprs <- t_v1_texprs[, t_v1_pred]
tc_vall_preproc <- preProcess(tc_vall_texprs, method = "center")
tc_vall_texprs <- predict(tc_vall_preproc, tc_vall_texprs)
tc_v1_preproc <- preProcess(tc_v1_texprs, method = "center")
tc_v1_texprs <- predict(tc_v1_preproc, tc_v1_texprs)
t_vall_preproc <- preProcess(t_vall_texprs, method = "center")
t_vall_texprs <- predict(t_vall_preproc, t_vall_texprs)
t_v1_preproc <- preProcess(t_v1_texprs, method = "center")
t_v1_texprs <- predict(t_v1_preproc, t_v1_texprs)
tc_vall_preproc <- preProcess(tc_vall_texprs, method = "corr", cutoff = 0.95)
tc_vall_texprs <- predict(tc_vall_preproc, tc_vall_texprs)
tc_v1_preproc <- preProcess(tc_v1_texprs, method = "corr", cutoff = 0.95)
tc_v1_texprs <- predict(tc_v1_preproc, tc_v1_texprs)
t_vall_preproc <- preProcess(t_vall_texprs, method = "corr", cutoff = 0.95)
t_vall_texprs <- predict(t_vall_preproc, t_vall_texprs)
t_v1_preproc <- preProcess(t_v1_texprs, method = "corr", cutoff = 0.95)
t_v1_texprs <- predict(t_v1_preproc, t_v1_texprs)
chosen_factors <- c("finaloutcome", "donor", "persistence", "visitnumber",
"selectionmethod", "typeofcells", "time")
tc_vall_meta <- pData(tc_vall_norm)[, chosen_factors]
tc_v1_meta <- pData(tc_v1_norm)[, chosen_factors]
t_vall_meta <- pData(t_vall_norm)[, chosen_factors]
t_v1_meta <- pData(t_v1_norm)[, chosen_factors]
ref_col <- "finaloutcome"
tc_vall_outcome <- as.factor(as.character(pData(tc_vall_norm)[[ref_col]]))
tc_v1_outcome <- as.factor(as.character(pData(tc_v1_norm)[[ref_col]]))
t_vall_outcome <- as.factor(as.character(pData(t_vall_norm)[[ref_col]]))
t_v1_outcome <- as.factor(as.character(pData(t_v1_norm)[[ref_col]]))
tc_vall_ml_df <- cbind.data.frame(tc_vall_outcome, as.data.frame(tc_vall_texprs))
tc_vall_ml_df[[ref_col]] <- as.factor(tc_vall_ml_df[["tc_vall_outcome"]])
tc_v1_ml_df <- cbind.data.frame(tc_v1_outcome, as.data.frame(tc_v1_texprs))
tc_v1_ml_df[[ref_col]] <- as.factor(tc_v1_ml_df[["tc_v1_outcome"]])
t_vall_ml_df <- cbind.data.frame(t_vall_outcome, as.data.frame(t_vall_texprs))
t_vall_ml_df[[ref_col]] <- as.factor(t_vall_ml_df[["t_vall_outcome"]])
t_v1_ml_df <- cbind.data.frame(t_v1_outcome, as.data.frame(t_v1_texprs))
t_v1_ml_df[[ref_col]] <- as.factor(t_v1_ml_df[["t_v1_outcome"]])11.3 Iterate Tumaco+Cali all visits
tc_vall_summary_xlsx <- glue("excel/tc_vall_ml_summary-v{ver}.xlsx")
tc_vall_knn <- classify_n_times(tc_vall_texprs, tc_vall_meta,
outcome_column = ref_col,
method = "knn", sampler = "cv")
written <- write_classifier_summary(tc_vall_knn, excel = tc_vall_summary_xlsx)
tc_vall_gb <- classify_n_times(tc_vall_texprs, tc_vall_meta,
outcome_column = ref_col,
method = "xgbTree", sampler = "cv")
written <- write_classifier_summary(tc_vall_gb, excel = written[["wb"]])
tc_vall_glm <- classify_n_times(tc_vall_texprs, tc_vall_meta,
outcome_column = ref_col,
method = "glmnet", sampler = "cv")
written <- write_classifier_summary(tc_vall_glm, excel = written[["wb"]])
tc_vall_rf <- classify_n_times(tc_vall_texprs, tc_vall_meta,
outcome_column = ref_col,
method = "ranger", sampler = "cv")
written <- write_classifier_summary(tc_vall_rf, excel = written[["wb"]])
openxlsx::saveWorkbook(written[["wb"]], file = tc_vall_summary_xlsx)11.4 Iterate Tumaco+Cali Visit 1
tc_v1_summary_xlsx <- glue("excel/tc_v1_ml_summary-v{ver}.xlsx")
tc_v1_knn <- classify_n_times(tc_v1_texprs, tc_v1_meta,
outcome_column = ref_col,
method = "knn", sampler = "cv")
written <- write_classifier_summary(tc_v1_knn, excel = tc_v1_summary_xlsx)
tc_v1_gb <- classify_n_times(tc_v1_texprs, tc_v1_meta,
outcome_column = ref_col,
method = "xgbTree", sampler = "cv")
written <- write_classifier_summary(tc_v1_gb, excel = written[["wb"]])
tc_v1_glm <- classify_n_times(tc_v1_texprs, tc_v1_meta,
outcome_column = ref_col,
method = "glmnet", sampler = "cv")
written <- write_classifier_summary(tc_v1_glm, excel = written[["wb"]])
tc_v1_rf <- classify_n_times(tc_v1_texprs, tc_v1_meta,
outcome_column = ref_col,
method = "ranger", sampler = "cv")
written <- write_classifier_summary(tc_v1_rf, excel = written[["wb"]])
openxlsx::saveWorkbook(written[["wb"]], file = tc_v1_summary_xlsx)11.5 Tumaco all visits
t_vall_summary_xlsx <- glue("excel/t_vall_ml_summary-v{ver}.xlsx")
t_vall_knn <- classify_n_times(t_vall_texprs, t_vall_meta,
outcome_column = ref_col,
method = "knn", sampler = "cv")
written <- write_classifier_summary(t_vall_knn, excel = t_vall_summary_xlsx)
t_vall_gb <- classify_n_times(t_vall_texprs, t_vall_meta,
outcome_column = ref_col,
method = "xgbTree", sampler = "cv")
written <- write_classifier_summary(t_vall_gb, excel = written[["wb"]])
t_vall_glm <- classify_n_times(t_vall_texprs, t_vall_meta,
outcome_column = ref_col,
method = "glmnet", sampler = "cv")
written <- write_classifier_summary(t_vall_glm, excel = written[["wb"]])
t_vall_rf <- classify_n_times(t_vall_texprs, t_vall_meta,
outcome_column = ref_col,
method = "ranger", sampler = "cv")
written <- write_classifier_summary(t_vall_rf, excel = written[["wb"]])
openxlsx::saveWorkbook(written[["wb"]], file = t_vall_summary_xlsx)11.6 Tumaco Visit 1
t_v1_summary_xlsx <- glue("excel/t_v1_ml_summary-v{ver}.xlsx")
t_v1_knn <- classify_n_times(t_v1_texprs, t_v1_meta,
outcome_column = ref_col,
method = "knn", sampler = "cv")
written <- write_classifier_summary(t_v1_knn, excel = t_v1_summary_xlsx)
t_v1_gb <- classify_n_times(t_v1_texprs, t_v1_meta,
outcome_column = ref_col,
method = "xgbTree", sampler = "cv")
written <- write_classifier_summary(t_v1_gb, excel = written[["wb"]])
t_v1_glm <- classify_n_times(t_v1_texprs, t_v1_meta,
outcome_column = ref_col,
method = "glmnet", sampler = "cv")
written <- write_classifier_summary(t_v1_glm, excel = written[["wb"]])
t_v1_rf <- classify_n_times(t_v1_texprs, t_v1_meta,
outcome_column = ref_col,
method = "ranger", sampler = "cv")
written <- write_classifier_summary(t_v1_rf, excel = written[["wb"]])
openxlsx::saveWorkbook(written[["wb"]], file = t_v1_summary_xlsx)Bibliography
Chen, Tianqi, and Carlos Guestrin. 2016. “XGBoost:
A Scalable Tree Boosting System.” In Proceedings
of the 22nd ACM SIGKDD International Conference on
Knowledge Discovery and Data Mining,
785–94. KDD ’16. New York, NY, USA: Association for
Computing Machinery. https://doi.org/10.1145/2939672.2939785.
Kuhn, Max. 2008. “Building Predictive Models in
R Using the Caret Package.” Journal
of Statistical Software 28 (5): 1–26. https://doi.org/10.18637/jss.v028.i05.
Wright, Marvin N., and Andreas Ziegler. 2017. “Ranger: A
Fast Implementation of Random Forests for High
Dimensional Data in C++ and R.”
Journal of Statistical Software 77 (March): 1–17. https://doi.org/10.18637/jss.v077.i01.
LS0tCnRpdGxlOiAiVE1SQzMgTUwgQ2xhc3NpZmljYXRpb24gb2Ygb3V0Y29tZTogYHIgU3lzLmdldGVudignVkVSU0lPTicpYCIKYXV0aG9yOiAiYXRiIGFiZWxld0BnbWFpbC5jb20iCmRhdGU6ICJgciBTeXMuRGF0ZSgpYCIKYmlibGlvZ3JhcGh5OiBhdGIuYmliCm91dHB1dDoKICBodG1sX2RvY3VtZW50OgogICAgY29kZV9kb3dubG9hZDogdHJ1ZQogICAgY29kZV9mb2xkaW5nOiBzaG93CiAgICBmaWdfY2FwdGlvbjogdHJ1ZQogICAgZmlnX2hlaWdodDogNwogICAgZmlnX3dpZHRoOiA3CiAgICBoaWdobGlnaHQ6IHplbmJ1cm4KICAgIGtlZXBfbWQ6IGZhbHNlCiAgICBtb2RlOiBzZWxmY29udGFpbmVkCiAgICBudW1iZXJfc2VjdGlvbnM6IHRydWUKICAgIHNlbGZfY29udGFpbmVkOiB0cnVlCiAgICB0aGVtZTogcmVhZGFibGUKICAgIHRvYzogdHJ1ZQogICAgdG9jX2Zsb2F0OgogICAgICBjb2xsYXBzZWQ6IGZhbHNlCiAgICAgIHNtb290aF9zY3JvbGw6IGZhbHNlCi0tLQoKPHN0eWxlIHR5cGU9InRleHQvY3NzIj4KYm9keSAubWFpbi1jb250YWluZXIgewogIG1heC13aWR0aDogMTYwMHB4Owp9CmJvZHksIHRkIHsKICBmb250LXNpemU6IDE2cHg7Cn0KY29kZS5yewogIGZvbnQtc2l6ZTogMTZweDsKfQpwcmUgewogIGZvbnQtc2l6ZTogMTZweAp9Cjwvc3R5bGU+CgpgYGB7ciwgaW5jbHVkZT1GQUxTRX0KbGlicmFyeShocGdsdG9vbHMpCmxpYnJhcnkoY2FyZXQpCmxpYnJhcnkoZHBseXIpCmxpYnJhcnkocFJPQykKbGlicmFyeShEQUxFWCkKbGlicmFyeShnbG1uZXQpCmxpYnJhcnkoZ2x1ZSkKbGlicmFyeShrZXJubGFiKQpsaWJyYXJ5KHJhbmdlcikKbGlicmFyeSh4Z2Jvb3N0KQpsaWJyYXJ5KGdnc3RhdHNwbG90KQoKa25pdHI6Om9wdHNfa25pdCRzZXQocHJvZ3Jlc3MgPSBUUlVFLCB2ZXJib3NlID0gVFJVRSwgd2lkdGggPSA5MCwgZWNobyA9IFRSVUUpCmtuaXRyOjpvcHRzX2NodW5rJHNldCgKICBlcnJvciA9IFRSVUUsIGZpZy53aWR0aCA9IDgsIGZpZy5oZWlnaHQgPSA4LCBmaWcucmV0aW5hID0gMiwKICBvdXQud2lkdGggPSAiMTAwJSIsIGRldiA9ICJwbmciLAogIGRldi5hcmdzID0gbGlzdChwbmcgPSBsaXN0KHR5cGUgPSAiY2Fpcm8tcG5nIikpKQpvbGRfb3B0aW9ucyA8LSBvcHRpb25zKGRpZ2l0cyA9IDQsIHN0cmluZ3NBc0ZhY3RvcnMgPSBGQUxTRSwga25pdHIuZHVwbGljYXRlLmxhYmVsID0gImFsbG93IikKZ2dwbG90Mjo6dGhlbWVfc2V0KGdncGxvdDI6OnRoZW1lX2J3KGJhc2Vfc2l6ZSA9IDEyKSkKdmVyIDwtIFN5cy5nZXRlbnYoIlZFUlNJT04iKQpydW5kYXRlIDwtIGZvcm1hdChTeXMuRGF0ZSgpLCBmb3JtYXQgPSAiJVklbSVkIikKCnJtZF9maWxlIDwtICIwNmNsYXNzaWZpZXJfaGlnaHZhci5SbWQiCnNhdmVmaWxlIDwtIGdzdWIocGF0dGVybiA9ICJcXC5SbWQiLCByZXBsYWNlID0gIlxcLnJkYVxcLnh6IiwgeCA9IHJtZF9maWxlKQpsb2FkZWQgPC0gbG9hZChmaWxlID0gZ2x1ZSgicmRhL3RtcmMzX2RhdGFfc3RydWN0dXJlcy12e3Zlcn0ucmRhIikpCmBgYAoKIyBDaGFuZ2Vsb2cKCiogMjAyNDA4OiBDb2xsYXBzaW5nIHRoZSB2YXJpb3VzIE1MIGRvY3VtZW50cyBiYWNrIGludG8gb25lIGluIHRoZQogIGhvcGVzIHRoYXQgaXQgZWFzZXMgdGhlIGRlZ3JlZSBvZiBjb25mdXNpb24uICBBbGwgb2YgdGhlIGV4cGxvcmF0b3J5CiAgYW5hbHlzZXMgYXJlIGF0IHRoZSB0b3AsIGF0IHRoZSBib3R0b20gYXJlIHRoZSBibG9ja3Mgd2hpY2ggYXJlIHVzZWQKICB0byBnZW5lcmF0ZSB0aGUgZmluYWwgdGFibGVzLCBidXQgdGhleSBhcmUgb25seSBydW5uYWJsZSBtYW51YWxseQogIGR1ZSB0byB0aW1lIGNvbnN0cmFpbnRzLgoqIDIwMjQwODogUmUtZW5hYmxlZCBnbG0sIGl0IHNlZW1zIHRvIGJlIHJ1bm5pbmcgZmFzdGVyIGFnYWluPwoqIDIwMjQwNjIwOiBEaXNhYmxlZCBnbG0gdHJhaW5pbmcvdGVzdGluZyBpbiB0aGUgY29udGFpbmVyLiAgRm9yCiAgcmVhc29ucyBJIGhhdmUgbm90IHlldCBmaWd1cmVkIG91dCBpdCB0YWtlcyB+IDEweCBsb25nZXIgKH4xMCBkYXlzKQogIGluIHRoZSBjb250YWluZXIgKG5vdGUsIHRoaXMgaXMgYSBuZXcgdGhpbmcsIGl0IHVzZWQgdG8gdGFrZSB+IDgKICBob3VycyBpbiB0aGUgY29udGFpbmVyIHdpdGhvdXQgZXJyb3IsIHNvIHNvbWV0aGluZyBjaGFuZ2VkIGluCiAgMjAyNDA1LTIwMjQwNiBjYXJldC9nbG1uZXQvc2luZ3VsYXJpdHkvZXRjKSB0byBjb21wbGV0ZTsgd2hpY2ggaXMgYQogIHByb2JsZW0gYmVjYXVzZSBteSBub3RlYm9va3MgYXJlIGF1dG8tcmVnZW5lcmF0ZWQgb24gYSAyeC93ZWVrIGJhc2lzCiAgKGlmIGNoYW5nZXMgb2NjdXIpIGFuZCBzbyBpdCBkb2Vzbid0IGZpbmlzaCBiZWZvcmUgYW5vdGhlciByb3VuZAogIHN0YXJ0cy4uLgoqIDIwMjMwOTogQWRkaW5nIGEgdmVyc2lvbiBvZiB0aGlzIHdpdGggb25seSB0aGUgVHVtYWNvIGRhdGEuICBJIGFtIGEKICBwcmV0dHkgYmlnIG5vbi1mYW4gb2YgZG9pbmcgdGhpczsgc28gSSBhbSBnb2luZyB0byBkbyBzbyBpbiB0aGUgbW9zdAogIGN1cnNvcnkgZmFzaGlvbiBwb3NzaWJsZSBieSBqdXN0IGNvcHlpbmcgdGhpcyBkb2N1bWVudCB0byBhbm90aGVyIGNvcHkgd2l0aAogIGEgc3VmZml4IG9mICd0dW1hY28nIGFuZCBjaGFuZ2luZyB0aGUgaW5pdGlhbCBkYXRhc3RydWN0dXJlIHRvIHRoZSBUdW1hY28tb25seQogIGRhdGFzZXQuCgoKIyBJbnRyb2R1Y3Rpb24KCkkgaGFkIHNvbWUgc3VjY2VzcyBpbiBjbGFzc2lmeWluZyB0aGUgVE1SQzIgc2FtcGxlcyBieSBzdHJhaW4gdmlhIE1MCmFuZCB3YW50IHRvIHRyeSBzb21ldGhpbmcgbW9yZSBkaWZmaWN1bHQuICBUaHVzIEkgd2lsbCB1c2UgdGhlCm5vcm1hbGl6ZWQgZ2VuZSBleHByZXNzaW9uIGRhdGEgYW5kIHRyeSBjbGFzc2lmeWluZyBpdCBieSBjdXJlL2ZhaWwuCgpBbGwgYW5hbHlzZXMgaW4gdGhpcyBkb2N1bWVudCBoZWF2aWx5IHVzZSBjYXJldAooQGt1aG5CdWlsZGluZ1ByZWRpY3RpdmVNb2RlbHMyMDA4KSBhbmQsIGFzIHBlciB0aGUgc2lzdGVyIFRNUkMyCmRvY3VtZW50LCBmb2xsb3cgcHJldHR5IGNsb3NlbHkgdGhlIHBhdGggZnJvbToKCmh0dHBzOi8vdG9wZXBvLmdpdGh1Yi5pby9jYXJldC9kYXRhLXNwbGl0dGluZy5odG1sI3NpbXBsZS1zcGxpdHRpbmctYmFzZWQtb24tdGhlLW91dGNvbWUKCmFuZAoKaHR0cHM6Ly9naXRodWIuY29tL2NvbXBnZW5vbXIvYm9vay9ibG9iL21hc3Rlci8wNS1zdXBlcnZpc2VkTGVhcm5pbmcuUm1kCgojIFN0YXJ0ZXIgZGF0YQoKSW4gdGhlIHN0cmFpbiBjbGFzc2lmaWVyIEkgdXNlZCBub3JtYWxpemVkIHZhcmlhbnRzLiAgSSBhbSB0aGlua2luZyB0bwp1c2Ugbm9ybWFsaXplZCBleHByZXNzaW9uIGhlcmUgYW5kIHRoZXJlZm9yZSBleHBsaWNpdGx5IGxpbWl0IG15c2VsZgp0byB+IDIwayB2YXJpYWJsZXMgKHNpZ25pZmljYW50bHkgZG93biBmcm9tIHRoZSAxLjZNKS4KCkluIGFkZGl0aW9uLCBjYXJldCBleHBlY3RzIHRoZSBkYXRhIGFzIChyb3dzID09IHNhbXBsZXMpIGFuZAooY29sdW1ucyA9PSB2YXJpYWJsZXMpIHdoZXJlIGVhY2ggZWxlbWVudCBpcyBvbmUgb2JzZXJ2YXRpb24uClRodXMgd2Ugd2lsbCBuZWVkIHRvIHRyYW5zcG9zZSB0aGUgZXhwcmVzc2lvbiBtYXRyaXguCgpgYGB7cn0KaW5wdXRfZGF0YSA8LSBzdWJzZXRfZXhwdCh0Y19jbGluaWNhbCwgc3Vic2V0PSJiYXRjaCE9J2Jpb3BzeSciKSAlPiUKICBub3JtYWxpemVfZXhwdCh0cmFuc2Zvcm0gPSAibG9nMiIsIGNvbnZlcnQgPSAiY3BtIikKCnJlZl9jb2wgPC0gImZpbmFsb3V0Y29tZSIKb3V0Y29tZV9mYWN0b3IgPC0gYXMuZmFjdG9yKGFzLmNoYXJhY3RlcihwRGF0YShpbnB1dF9kYXRhKVtbcmVmX2NvbF1dKSkKY29tcGFyaXNvbl9uIDwtIDIwMAoKIyMgU2V0dGluZyB0aGlzIHVwIGluIGNhc2UgTmFqaWIgYW5kIE1hcmlhIEFkZWxhaWRhIHJlcXVlc3QgYSBUdW1hY28gc2V0IG9mIGV4cGxvcmF0b3J5IGFuYWx5c2VzLgp0X2lucHV0X2RhdGEgPC0gc3Vic2V0X2V4cHQodGNfY2xpbmljYWwsIHN1YnNldD0iYmF0Y2ghPSdiaW9wc3knIikgJT4lCiAgbm9ybWFsaXplX2V4cHQodHJhbnNmb3JtID0gImxvZzIiLCBjb252ZXJ0ID0gImNwbSIpCnRfb3V0Y29tZV9mYWN0b3IgPC0gYXMuZmFjdG9yKGFzLmNoYXJhY3RlcihwRGF0YSh0X2lucHV0X2RhdGEpW1tyZWZfY29sXV0pKQpgYGAKCiMgQ29tcGFyZSBUb3BOIGltcG9ydGFudCBnZW5lcyB0byBERSBlbmVzCgpHaXZlbiB0aGF0IHdlIGhhdmUgc2VwYXJhdGVkIHRoZSB2YXJpb3VzIGFuYWx5c2VzLCBpdCB3aWxsIHRha2UgbWUgYQptaW51dGUgdG8gZmlndXJlIG91dCB3aGVyZSBJIHNhdmVkIHRoZSByZWxldmFudCBkaWZmZXJlbnRpYWwKZXhwcmVzc2lvbiBhbmFseXNpcy4gIEkgZG8gbm90IGFjdHVhbGx5IHNhdmUgdGhlIHZhcmlvdXMgREUgcmVzdWx0cyB0bwpyZGEgZmlsZXMgYnkgZGVmYXVsdCwgaW5zdGVhZCBvcHRpbmcgdG8gc2VuZCB0aGVtIHRvIHhsc3ggZmlsZXMgdG8Kc2hhcmUuICBSZWNhbGwgaWYgeW91IHdpbGwsIHRoYXQgdGhlIGRhdGEgdGhhdCBJIHRoaW5rIG1pZ2h0IGJlIHVzZWQKZm9yIHRoZSBwYXBlciBhbHNvIGRvZXMgbm90IGdvIGludG8gdGhlIGRlZmF1bHQgZXhjZWwgZGlyZWN0b3J5IGJ1dAppbnN0ZWFkIG1pcnJvcnMgdGhlIGJveCBvcmdhbml6YXRpb24gc2NoZW1lLgoKVGh1cywgSSB0aGluayB0aGUgbW9zdCByZWxldmFudCBmaWxlIGlzOgoiYW5hbHlzZXMvM19jYWxpX2FuZF90dW1hY28vREVfQ3VyZV92c19GYWlsL0FsbF9TYW1wbGVzL3RjX2NmX2NsaW5pY2FsX3RhYmxlX3N2YS12MjAyMjA3Lnhsc3giCgphbmQKCiJhbmFseXNlcy80X3R1bWFjby9ERV9DdXJlX3ZzX0ZhaWwvQWxsX1NhbXBsZXMvdF9jZl9jbGluaWNhbF90YWJsZV9zdmEtdjIwMjIwNy54bHN4IgoKTm90ZSwgdGhpcyB4bHN4IGZpbGUgY29udGFpbnMgdGhyZWUgcHJpbWFyeSB3b3Jrc2hlZXRzLCB2MWNmLCB2MmNmLAp2M2NmIHdoaWNoIGNvbXByaXNlIGFsbCBzYW1wbGVzIGFjcm9zcyBjZWxsIHR5cGVzIGZvciBlYWNoIHZpc2l0IGFuZApjb21wYXJlcyBjdXJlIGFuZCBmYWlsLiAgV2hlbiBJIGNob29zZSBzaGVldCAjMiBpbiB0aGUgZm9sbG93aW5nIGJsb2NrCkkgYW0gZXhwbGljaXRseSBhc2tpbmcgZm9yIG9ubHkgdGhlIHYxIGN1cmUvZmFpbCByZXN1bHQuCgojIyBPcGVuaW5nIHRoZSByZWxldmFudCBERSB0YWJsZQoKYGBge3J9CmlucHV0X3hsc3ggPC0gZ2x1ZSgiYW5hbHlzZXMvM19jYWxpX2FuZF90dW1hY28vREVfQ3VyZV9GYWlsL0NsaW5pY2FsX1NhbXBsZXMvdGNfY2xpbmljYWxfY2Zfc2lnX3N2YS12e3Zlcn0ueGxzeCIpCmFsbF9kZV9jZiA8LSBvcGVueGxzeDo6cmVhZFdvcmtib29rKGlucHV0X3hsc3gsIHNoZWV0ID0gMywgc3RhcnRSb3cgPSAyKQpyb3duYW1lcyhhbGxfZGVfY2YpIDwtIGFsbF9kZV9jZltbInJvdy5uYW1lcyJdXQphbGxfZGVfY2ZbWyJyb3cubmFtZXMiXV0gPC0gTlVMTApkZXNlcV9kZV9jZl91cCA8LSBhbGxfZGVfY2ZbLCBjKCJkZXNlcV9sb2dmYyIsICJkZXNlcV9hZGpwIiwgImRlc2VxX2Jhc2VtZWFuIiwgImRlc2VxX2xmY3NlIildCmFsbF9kZV9jZiA8LSBvcGVueGxzeDo6cmVhZFdvcmtib29rKGlucHV0X3hsc3gsIHNoZWV0ID0gNCwgc3RhcnRSb3cgPSAyKQpyb3duYW1lcyhhbGxfZGVfY2YpIDwtIGFsbF9kZV9jZltbInJvdy5uYW1lcyJdXQphbGxfZGVfY2ZbWyJyb3cubmFtZXMiXV0gPC0gTlVMTApkZXNlcV9kZV9jZl9kb3duIDwtIGFsbF9kZV9jZlssIGMoImRlc2VxX2xvZ2ZjIiwgImRlc2VxX2FkanAiLCAiZGVzZXFfYmFzZW1lYW4iLCAiZGVzZXFfbGZjc2UiKV0KZGVzZXFfZGVfY2YgPC0gcmJpbmQuZGF0YS5mcmFtZShkZXNlcV9kZV9jZl91cCwgZGVzZXFfZGVfY2ZfZG93bikKCnR1bWFjb194bHN4IDwtIGdsdWUoImFuYWx5c2VzLzRfdHVtYWNvL0RFX0N1cmVfRmFpbC90X2FsbF92aXNpdGNmX3NpZ19zdmEtdnt2ZXJ9Lnhsc3giKQp0X2RlX2NmIDwtIG9wZW54bHN4OjpyZWFkV29ya2Jvb2sodHVtYWNvX3hsc3gsIHNoZWV0ID0gMywgc3RhcnRSb3cgPSAyKQpyb3duYW1lcyh0X2RlX2NmKSA8LSB0X2RlX2NmW1sicm93Lm5hbWVzIl1dCnRfZGVfY2ZbWyJyb3cubmFtZXMiXV0gPC0gTlVMTAp0X2RlX2NmX3VwIDwtIHRfZGVfY2ZbLCBjKCJkZXNlcV9sb2dmYyIsICJkZXNlcV9hZGpwIiwgImRlc2VxX2Jhc2VtZWFuIiwgImRlc2VxX2xmY3NlIildCnRfZGVfY2YgPC0gb3Blbnhsc3g6OnJlYWRXb3JrYm9vayh0dW1hY29feGxzeCwgc2hlZXQgPSA0LCBzdGFydFJvdyA9IDIpCnJvd25hbWVzKHRfZGVfY2YpIDwtIHRfZGVfY2ZbWyJyb3cubmFtZXMiXV0KdF9kZV9jZltbInJvdy5uYW1lcyJdXSA8LSBOVUxMCnRfZGVfY2ZfZG93biA8LSB0X2RlX2NmWywgYygiZGVzZXFfbG9nZmMiLCAiZGVzZXFfYWRqcCIsICJkZXNlcV9iYXNlbWVhbiIsICJkZXNlcV9sZmNzZSIpXQp0X2RlX2NmIDwtIHJiaW5kLmRhdGEuZnJhbWUodF9kZV9jZl91cCwgdF9kZV9jZl9kb3duKQp0b3BfYm90dG9tX2lkcyA8LSByb3duYW1lcyh0X2RlX2NmKQpgYGAKCiMjIEEgZnVuY3Rpb24gdG8gaGVscCBjb21wYXJlIG1ldGhvZHMgdG8gREVTZXEyCgpUaGUgZm9sbG93aW5nIGJsb2NrIGNvbnRhaW5zIGEgZmV3IGxpc3RzIHdoaWNoIHdpbGwgaG9sZCBjb21wYXJpc29ucwpiZXR3ZWVuIERFU2VxMiBhbmQgdGhlIHZhcmlvdXMgbWV0aG9kcyB3aGljaCBmb2xsb3csIHNvbWUgcGFyYW1ldGVycywKYW5kIGEgZnVuY3Rpb24gdG8gY29tcGFyZSB0aGUgbWV0aG9kcyBhbmQgREVTZXEuCgpQcmVzdW1hYmx5IERFU2VxIGFuZCB0aGUgbW9kZWxzIHNob3VsZCBiZSByZXNwb25kaW5nIHRvIHZhcmlhbmNlIGluCnRoZSBkYXRhLCBmb3Igd2hpY2ggSSB0aGluayB0aGUgbG9nRkMgdmFsdWVzLCBwLXZhbHVlcywgbWVhbiB2YWx1ZXMsCm9yIHN0YW5kYXJkIGVycm9ycyBhcmUgdGhlIG1vc3QgbGlrZWx5IHByb3hpZXMgdG8gd2hpY2ggSSBoYXZlIGVhc3kKYWNjZXNzLiAgU28sIGxldCB1cyBwdWxsIHRoZSB0b3AvYm90dG9tIG4gZ2VuZXMgdmlzIGEgdmlzIGVhY2ggb2YKdGhvc2UgY2F0ZWdvcmllcyBhbmQgc2VlIHdoYXQgaGFwcGVucz8KCkhlcmUgaXMgYSBsaXR0bGUgZnVuY3Rpb24gdG8gY29tcGFyZSBhIE1MIHJlc3VsdCB0byB0aGUgYWJvdmUgREUKdGFibGUuICBJdCBzaG91bGQgcHJvdmlkZSB0aGUgaW1wb3J0YW5jZSBwbG90LCBhIHZlbm4gb2YgdGhlIHR3bwptZXRob2RzLCBhbmQgdGhlIGdlbmUgc2V0cyAod2l0aCBhbm5vdGF0aW9ucykgZm9yIHRoZSB1bmlxdWUvc2hhcmVkCmdlbmVzIGJldHdlZW4gdGhpcyBNTCBtZXRob2QgYW5kIHRoZSBERSB0YWJsZS4gIE5vdGUsIHRoaXMgaXMganVzdAp1c2luZyB0aGUgaGlnaGVzdC9sb3dlc3QgbG9nRkMgdmFsdWVzIGFuZCBpZ25vcmluZyB0aGUgcC12YWx1ZSBmb3Igbm93LgoKYGBge3J9CmNvbXBhcmlzb25fbGZjIDwtIGxpc3QoKQpjb21wYXJpc29uX2FkanAgPC0gbGlzdCgpCmNvbXBhcmlzb25fd2djbmEgPC0gbGlzdCgpCmFubm90YXRpb25zIDwtIGZEYXRhKHRfbW9ub2N5dGVzKSAgIyMgQXJiaXRyYXJpbHkgdGFrZW4KCmNvbXBhcmVfbWxfZGUgPC0gZnVuY3Rpb24oaW1wb3J0YW5jZSwgc2lnX3RhYmxlLCBhbm5vdGF0aW9ucywgbiA9IGNvbXBhcmlzb25fbiwgcGxvdF9uID0gMTUsCiAgICAgICAgICAgICAgICAgICAgICAgICAgYWNjb3JkaW5nX3RvID0gImRlc2VxX2xvZ2ZjIiwgY29sdW1uID0gIk92ZXJhbGwiKSB7CiAgbWxfZGYgPC0gaW1wb3J0YW5jZVtbImltcG9ydGFuY2UiXV0gJT4lCiAgICBhcnJhbmdlKGRlc2MoISFzeW0oY29sdW1uKSkpCiAgaW1wb3J0YW5jZV9wbG90IDwtIHBsb3QoaW1wb3J0YW5jZSwgdG9wID0gcGxvdF9uKQogIHRvcF9tbCA8LSBoZWFkKHJvd25hbWVzKG1sX2RmKSwgbiA9IG4pCiAgZGVfZGYgPC0gc2lnX3RhYmxlICU+JQogICAgYXJyYW5nZShkZXNjKGRlc2VxX2xvZ2ZjKSkKICB0b3BfZGUgPC0gaGVhZChyb3duYW1lcyhkZV9kZiksIG4gPSBuKQogIGJvdHRvbV9kZSA8LSB0YWlsKHJvd25hbWVzKGRlX2RmKSwgbiA9IG4pCiAgdG9wX2JvdHRvbV9pZHMgPC0gYyh0b3BfZGUsIGJvdHRvbV9kZSkKICBjb21wYXJpc29uIDwtIGxpc3QoImRlIiA9IHRvcF9ib3R0b21faWRzLCAibWwiID0gdG9wX21sKQogIGNvbXBhcmlzb25fdmVubiA8LSBWZW5uZXJhYmxlOjpWZW5uKGNvbXBhcmlzb24pCiAgc2hhcmVkX2lkcyA8LSBjb21wYXJpc29uX3Zlbm5ASW50ZXJzZWN0aW9uU2V0c1siMTEiXVtbMV1dCiAgc2hhcmVkX2Fubm90IDwtIGFubm90YXRpb25zW3NoYXJlZF9pZHMsIF0KICBkZV9pZHMgPC0gY29tcGFyaXNvbl92ZW5uQEludGVyc2VjdGlvblNldHNbIjEwIl1bWzFdXQogIG1sX2lkcyA8LSBjb21wYXJpc29uX3Zlbm5ASW50ZXJzZWN0aW9uU2V0c1siMDEiXVtbMV1dCiAgb25seV9kZSA8LSBhbm5vdGF0aW9uc1tkZV9pZHMsIF0KICBvbmx5X21sIDwtIGFubm90YXRpb25zW21sX2lkcywgXQogIGNvbXBhcmlzb25fcGxvdCA8LSBWZW5uZXJhYmxlOjpwbG90KGNvbXBhcmlzb25fdmVubiwgZG9XZWlnaHRzID0gRkFMU0UsIHR5cGUgPSAiY2lyY2xlcyIpCiAgcmV0bGlzdCA8LSBsaXN0KAogICAgImltcG9ydGFuY2VfcGxvdCIgPSBpbXBvcnRhbmNlX3Bsb3QsCiAgICAiY29tcGFyaXNvbiIgPSBjb21wYXJpc29uLAogICAgImNvbXBhcmlzb25fdmVubiIgPSBjb21wYXJpc29uX3Zlbm4sCiAgICAiY29tcGFyaXNvbl9wbG90IiA9IGNvbXBhcmlzb25fcGxvdCwKICAgICJzaGFyZWQiID0gc2hhcmVkX2Fubm90LAogICAgIm9ubHlfZGUiID0gb25seV9kZSwKICAgICJvbmx5X21sIiA9IG9ubHlfbWwpCiAgcmV0dXJuKHJldGxpc3QpCn0KYGBgCgojIEZpbHRlcmluZwoKVGhlIE1MIHRleHQgSSBhbSByZWFkaW5nIHByb3ZpZGUgc29tZSBuZWF0IGV4YW1wbGVzIGZvciBob3cgb25lIG1pZ2h0CmZpbHRlciB0aGUgZGF0YSB0byBtYWtlIGl0IG1vcmUgc3VpdGFibGUgZm9yIG1vZGVsIGNyZWF0aW9uLgoKIyMgTmVhciB6ZXJvIHZhcmlhbmNlLCBvciBnZW5lZmlsdGVyJ3MgY3YKClRoZSBmaXJzdCBmaWx0ZXIgSSB3YXMgaW50cm9kdWNlZCB0byBpcyBxdWl0ZSBmYW1pbGlhciBmcm9tIG91cgpzZXF1ZW5jaW5nIGRhdGEsIHRoZSByZW1vdmFsIG9mIGZlYXR1cmVzIHdpdGggbmVhci16ZXJvLXZhcmlhbmNlLgpJbmRlZWQsIEkgYW0gcHJldHR5IGNlcnRhaW4gdGhhdCBub3JtYWxpemVfZXhwdCgpIGNvdWxkIGRvIHRoaXMKZXF1aXZhbGVudGx5IGFuZCBzaWduaWZpY2FudGx5IGZhc3RlciB0aGFuIGNhcmV0OjpwcmVQcm9jZXNzKCkuCgpgYGB7cn0KbnJvdyhleHBycyhpbnB1dF9kYXRhKSkKc3lzdGVtLnRpbWUoewogIGVxdWl2YWxlbnQgPC0gbm9ybWFsaXplX2V4cHQoaW5wdXRfZGF0YSwgZmlsdGVyID0gImN2IiwgY3ZfbWluID0gMC4xKQp9KQpkaW0oZXhwcnMoZXF1aXZhbGVudCkpCgojIyBHaXZlbiB0aGlzIGxhcmdlIGFtb3VudCBvZiBkYXRhLCB0aGlzIHN0ZXAgaXMgc2xvdywgdGFraW5nID4gMTAgbWludXRlcy4KIyMgWWVhaCBzZXJpb3VzbHksIHRoZSBmb2xsb3dpbmcgdGhyZWUgbGluZXMgZ2V0IDE2LDcyMyBnZW5lcyBpbiAxMCBtaW51dGVzIHdoaWxlCiMjIHRoZSBub3JtYWxpemVfZXhwdCgpIGNhbGwgYWJvdmUgZ2V0cyAxNiw3NDkgZ2VuZXMgaW4gMi40IHNlY29uZHMuCiNzeXN0ZW0udGltZSh7CiMgIG56diA8LSBwcmVQcm9jZXNzKHRleHBycywgbWV0aG9kPSJuenYiLCB1bmlxdWVDdXQ9MTUpCiMgIG56dl90ZXhwcnMgPC0gcHJlZGljdChuenYsIHRleHBycykKIyAgZGltKG56dl90ZXhwcnMpCiN9Cm56dl90ZXhwcnMgPC0gdChleHBycyhlcXVpdmFsZW50KSkKYGBgCgojIyBGaWx0ZXJpbmcgdG8gdGhlIGhpZ2hlc3Qgc3RhbmRhcmQgZGV2aWF0aW9uIHZhcmlhYmxlcwoKSSB0aGluayBJIGFtIGEgYml0IGNvbmZ1c2VkIGJ5IHRoaXMgZmlsdGVyLCBvbmUgd291bGQgdGhpbmsgdGhhdCB0aGUKbnp2IGZpbHRlciBhYm92ZSwgaWYgYXBwbGllZCBjb3JyZWN0bHksIHNob3VsZCBnaXZlIHlvdSBleGFjdGx5IHRoaXMuCgpGb3IgdGhlIG1vbWVudCwgSSBhbSBleGNsdWRpbmcgdGhlIGZvbGxvd2luZyBibG9jayBpbiBvcmRlciB0byBzZWUgaG93Cm11Y2ggdGltZS9tZW1vcnkga2VlcGluZyB0aGVzZSB2YXJpYWJsZXMgY29zdHMuICBJZiBJIHJlY2FsbCBwcm9wZXJseSwKdGhlIG1vZGVsIG9mIHRoZSB0b3AtMmsgdmFyaWFudCBwb3NpdGlvbnMgY29zdCB+IDEtNEcgb2YgbWVtb3J5LiAgSQpob3BlIHRoYXQgdGhpcyBzY2FsZXMgbGluZWFybHksIGJ1dCBJIGFtIHRoaW5raW5nIGl0IG1pZ2h0IG5vdC4KCmBgYHtyfQpzdGFuZGFyZF9kZXZzIDwtIGFwcGx5KG56dl90ZXhwcnMsIDIsIHNkKQp0b3BfcHJlZGljdG9ycyA8LSBvcmRlcihzdGFuZGFyZF9kZXZzLCBkZWNyZWFzaW5nID0gVFJVRSlbMTozMDAwXQpuenZfdGV4cHJzIDwtIG56dl90ZXhwcnNbLCB0b3BfcHJlZGljdG9yc10KYGBgCgojIyBDZW50ZXIgdGhlIGRhdGEKCkkgdGhpbmsgY2VudGVyaW5nIG1heSBub3QgYmUgbmVlZGVkIGZvciB0aGlzIGRhdGEsIGJ1dCBoZXJlIGlzIGhvdzoKCmBgYHtyfQpuenZfY2VudGVyIDwtIHByZVByb2Nlc3Mobnp2X3RleHBycywgbWV0aG9kID0gImNlbnRlciIpCm56dl90ZXhwcnMgPC0gcHJlZGljdChuenZfY2VudGVyLCBuenZfdGV4cHJzKQpgYGAKCiMjIERyb3AgY29ycmVsYXRlZAoKVGhpcyBpcyBhIGZpbHRlciB3aGljaCBkb2VzIG5vdCBjb3JyZXNwb25kIHRvIGFueSBvZiB0aG9zZSB3ZSB1c2UgaW4Kc2VxdWVuY2luZyBkYXRhIGJlY2F1c2UgZ2VuZXMgd2hpY2ggYXJlIGhpZ2hseSBjb3JyZWxhdGVkIGFyZQpsaWtlbHkgdG8gYmUgb2YgaW1tZWRpYXRlIGludGVyZXN0LgoKSW4gdGhlIHNhbWUgZmFzaGlvbiwgSSB3YW50IHRvIGxlYXZlIHRoaXMgb2ZmIGJlY2F1c2UgbGF0ZXIKYXBwbGljYXRpb25zIG9mIHRoaXMgbW9kZWwgd2lsbCBpbmNsdWRlIGxvdyBjb3ZlcmFnZSBzYW1wbGVzIHdoaWNoIG1heQpub3QgaGF2ZSBldmVyeSB2YXJpYW50IHJlcHJlc2VudGVkLgoKYGBge3J9CiMjIFRoaXMgc3RlcCB0YWtlcyBhIHdoaWxlLi4uCnN5c3RlbS50aW1lKHsKICBuenZfY29ycmVsYXRlZCA8LSBwcmVQcm9jZXNzKG56dl90ZXhwcnMsIG1ldGhvZCA9ICJjb3JyIiwgY3V0b2ZmID0gMC45NSkKICBuenZfdW5jb3JyIDwtIHByZWRpY3Qobnp2X2NvcnJlbGF0ZWQsIG56dl90ZXhwcnMpCn0pCmRpbShuenZfdW5jb3JyKQpgYGAKCiMgTWVyZ2UgdGhlIGFwcHJvcHJpYXRlIG1ldGFkYXRhCgpUaGVyZSBhcmUgYSBmZXcgbWV0YWRhdGEgZmFjdG9ycyB3aGljaCBtaWdodCBwcm92ZSBvZiBpbnRlcmVzdCBmb3IKY2xhc3NpZmljYXRpb24uICBUaGUgbW9zdCBvYnZpb3VzIGFyZSBvZiBjb3Vyc2Ugb3V0Y29tZSwgY2xpbmljLApkb25vciwgdmlzaXQsIGNlbGx0eXBlLiAgSSBhbSwgZm9yIHRoZSBtb21lbnQsIG9ubHkgbGlrZWx5IHRvIGZvY3VzIG9uCm91dGNvbWUuICBBRkFJQ1QgSSBjYW4gb25seSBpbmNsdWRlIG9uZSBvZiB0aGVzZSBhdCBhIHRpbWUgaW4gdGhlCmRhdGEsIHdoaWNoIGlzIGEgc2hhbWUuCgpgYGB7cn0KaW50ZXJlc3RpbmdfbWV0YSA8LSBwRGF0YShpbnB1dF9kYXRhKVssIGMoImZpbmFsb3V0Y29tZSIsICJkb25vciIsICJwZXJzaXN0ZW5jZSIsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICJ2aXNpdG51bWJlciIsICJzZWxlY3Rpb25tZXRob2QiLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAidHlwZW9mY2VsbHMiLCAidGltZSIpXQoKbWxfZGYgPC0gYXMuZGF0YS5mcmFtZShjYmluZChvdXRjb21lX2ZhY3RvciwgYXMuZGF0YS5mcmFtZShuenZfdW5jb3JyKSkpCm1sX2RmW1sib3V0Y29tZV9mYWN0b3IiXV0gPC0gYXMuZmFjdG9yKG1sX2RmW1sib3V0Y29tZV9mYWN0b3IiXV0pCmRpbShtbF9kZikKYGBgCgojIFNwbGl0IHRoZSBkYXRhIGludG8gdHJhaW5pbmcvdGVzdGluZwoKY2FyZXQgcHJvdmlkZXMgbmljZSBmdW5jdGlvbmFsaXR5IGZvciBzcGxpdHRpbmcgdXAgdGhlIGRhdGEuICBJCnN1c3BlY3QgdGhlcmUgYXJlIG1hbnkgbW9yZSBmdW4ga25vYnMgSSBjYW4gcGxheSB3aXRoIGZvciBpbnN0YW5jZXMKd2hlcmUgSSBuZWVkIHRvIGV4Y2x1ZGUgc29tZSBsZXZlbHMgb2YgYSBmYWN0b3IgYW5kIHN1Y2guICBJbiB0aGlzCmNhc2UgSSBqdXN0IHdhbnQgdG8gc3BsaXQgYnkgb3V0Y29tZS4KCiMjIFZpYSBkYXRhIHNwbGl0dGluZwoKYGBge3J9Cm1sX2RmIDwtIGFzLmRhdGEuZnJhbWUoY2JpbmQob3V0Y29tZV9mYWN0b3IsIGFzLmRhdGEuZnJhbWUobnp2X3VuY29ycikpKQoKZGF0YXNldHMgPC0gY3JlYXRlX3BhcnRpdGlvbnMobnp2X3VuY29yciwgaW50ZXJlc3RpbmdfbWV0YSwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgb3V0Y29tZV9mYWN0b3IgPSBvdXRjb21lX2ZhY3RvcikKYGBgCgojIyBWaWEgc2FtcGxpbmcKClRoZXJlIGFyZSBhIGZldyBsaWtlbHkgc2FtcGxpbmcgbWV0aG9kczogY3Jvc3MtdmFsaWRhdGlvbiwKYm9vdHN0cmFwcGluZywgYW5kIGphY2trbmlmaW5nLiAgSSB3aWxsIHRyeSB0aG9zZSBvdXQgbGF0ZXIuCgojIFRyeSBvdXQgdHJhaW5pbmcgYW5kIHByZWRpY3Rpb24gbWV0aG9kcwoKTXkgZ29hbHMgZnJvbSBoZXJlIG9uIHdpbGwgYmUgdG8gZ2V0IHRoZSBiZWdpbm5pbmdzIG9mIGEgc2Vuc2Ugb2YgdGhlCnZhcmlvdXMgbWV0aG9kcyBJIGNhbiB1c2UgdG8gY3JlYXRlIHRoZSBtb2RlbHMgZnJvbSB0aGUgdHJhaW5pbmcgZGF0YQphbmQgcHJlZGljdCB0aGUgb3V0Y29tZSBvbiB0aGUgdGVzdCBkYXRhLiAgSSBhbSBob3BpbmcgYWxzbyB0byBwaWNrIHVwCnNvbWUgaWRlYSBvZiB3aGF0IHRoZSB2YXJpb3VzIGFyZ3VtZW50cyBtZWFuIHdoaWxlIEkgYW0gYXQgaXQuCgojIyBUcnkgb3V0IEtOTgoKay1uZWFyZXN0IG5laWdoYm9ycyBpcyBzb21ld2hhdCBzaW1pbGFyIHRvIGEga21lYW5zIGVzdGltYXRlLiAgVGh1cwp0aGUgcHJpbWFyeSBhcmd1bWVudCBpcyAnaycKCiMjIyBNb2RlbCBjcmVhdGlvbiBhbmQgcGVyZm9ybWFuY2UKCmBgYHtyfQpzcGxpdCA8LSAxCnRyYWluX2FsbCA8LSBkYXRhc2V0c1tbInRyYWluZXJzIl1dW1tzcGxpdF1dCnRyYWluX2RmIDwtIGRhdGFzZXRzW1sidHJhaW5lcnNfc3RyaXBwZWQiXV1bW3NwbGl0XV0KdHJhaW5faWR4IDwtIGRhdGFzZXRzW1sidHJhaW5faWR4Il1dW1tzcGxpdF1dCnRyYWluX291dGNvbWVzIDwtIGRhdGFzZXRzW1sidHJhaW5lcl9vdXRjb21lcyJdXVtbc3BsaXRdXQp0ZXN0X2RmIDwtIGRhdGFzZXRzW1sidGVzdGVycyJdXVtbc3BsaXRdXQp0ZXN0X2lkeCA8LSBkYXRhc2V0c1tbInRlc3RfaWR4Il1dW1tzcGxpdF1dCnRlc3Rfb3V0Y29tZXMgPC0gZGF0YXNldHNbWyJ0ZXN0ZXJfb3V0Y29tZXMiXV1bW3NwbGl0XV0KCmtubl9maXQgPC0ga25uMyh4ID0gdHJhaW5fZGYsCiAgICAgICAgICAgICAgICB5ID0gdHJhaW5fb3V0Y29tZXMsCiAgICAgICAgICAgICAgICBrID0gMykKa25uX3ByZWRpY3RfdHJhaW5lZCA8LSBwcmVkaWN0KGtubl9maXQsIHRyYWluX2RmLCB0eXBlID0gInByb2IiKQoKa25uX3RyYWluX2V2YWx1YXRlZCA8LSBzZWxmX2V2YWx1YXRlX21vZGVsKGtubl9wcmVkaWN0X3RyYWluZWQsIGRhdGFzZXRzLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgd2hpY2ggPSBzcGxpdCwgdHlwZSA9ICJ0cmFpbiIpCmtubl90cmFpbl9ldmFsdWF0ZWQKYGBgCgpBcyB0aGUgY29uZnVzaW9uIG1hdHJpeCBzaG93cywgdGhpcyBmYWlsZWQgZm9yIGEgZmV3IHNhbXBsZXMuICBQZXJoYXBzCmxldCB1cyBjaGFuZ2UgayBhbmQgc2VlIGlmIGl0IGltcHJvdmVzLgoKSGVyZSBpcyBhIHRhYmxlIG9mIGZhc2UgcG9zaXRpdmVzL25lZ2F0aXZlcyBmb3IgYSBmZXcgdmFsdWVzIG9mICdrJywKaW4gdGhpcyBjb250ZXh0IGEgZmFsc2UgcG9zaXRpdmUgaXMgY2FsbGluZyBhIGtub3duIGN1cmUgYXMgYSBmYWlsdXJlCmFuZCBmYWxzZSBuZWdhdGl2ZSBpcyBjYWxsaW5nIGEga25vd24gZmFpbHVyZSBhcyBhIGN1cmUuCgp8LS0tfC0tLXwtLS18CnxrICB8ZnAgfGZuIHwKfDIgIHwwICB8OCAgfAp8MyAgfDUgIHw1ICB8Cnw0ICB8OCAgfDkgIHwKfDUgIHwxMSB8NyAgfAp8NiAgfDE1IHw4ICB8CgpOb3RlOiB0aGlzIGRlcGVuZHMgb24gdGhlIGx1Y2sgb2YgcmFuZCgpLCBzbyB0aGUgYWJvdmUgbnVtYmVycyBzaGlmdAptb2RlcmF0ZWx5IGZyb20gb25lIHJ1biB0byB0aGUgbmV4dC4gIFRodXMgSSB0aGluayBJIHdpbGwganVzdCB1c2UgMgpvciAzLgoKYGBge3J9Cmtubl9maXQyIDwtIGtubjMoeCA9IHRyYWluX2RmLAogICAgICAgICAgICAgICAgeSA9IHRyYWluX291dGNvbWVzLAogICAgICAgICAgICAgICAgayA9IDUpCmtubl9wcmVkaWN0X3RyYWluZWQyIDwtIHByZWRpY3Qoa25uX2ZpdDIsIHRyYWluX2RmLCB0eXBlID0gInByb2IiKQoKa25uX3RyYWluX2V2YWx1YXRlZDIgPC0gc2VsZl9ldmFsdWF0ZV9tb2RlbChrbm5fcHJlZGljdF90cmFpbmVkMiwgZGF0YXNldHMsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgd2hpY2ggPSBzcGxpdCwgdHlwZSA9ICJ0cmFpbiIpCmtubl90cmFpbl9ldmFsdWF0ZWQyCmBgYAoKIyMjIFByZWRpY3QgdGhlIHJlc3Qgb2YgdGhlIGRhdGEgd2l0aCB0aGlzIG1vZGVsLgoKYGBge3J9Cmtubl9wcmVkaWN0X3Rlc3QgPC0gcHJlZGljdChrbm5fZml0LCB0ZXN0X2RmKQoKa25uX3Rlc3RfZXZhbHVhdGVkIDwtIHNlbGZfZXZhbHVhdGVfbW9kZWwoa25uX3ByZWRpY3RfdGVzdCwgZGF0YXNldHMsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB3aGljaCA9IHNwbGl0LCB0eXBlID0gInRlc3QiKQprbm5fdGVzdF9ldmFsdWF0ZWQKCmtubl9wcmVkaWN0X3Rlc3QyIDwtIHByZWRpY3Qoa25uX2ZpdDIsIHRlc3RfZGYpCmtubl90ZXN0X2V2YWx1YXRlZDIgPC0gc2VsZl9ldmFsdWF0ZV9tb2RlbChrbm5fcHJlZGljdF90ZXN0MiwgZGF0YXNldHMsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB3aGljaCA9IHNwbGl0LCB0eXBlID0gInRlc3QiKQprbm5fdGVzdF9ldmFsdWF0ZWQyCmBgYAoKIyMgUGVyZm9ybSBjcm9zcy12YWxpZGF0aW9uIHRvIGVzdGltYXRlIGsKClRoZSBjcm9zcyB2YWxpZGF0aW9uIG1ldGhvZCBvZiByZXBlYXRlZCBzYW1wbGluZyB0aGUgZGF0YSBpcyBhbGwgZG9uZQp3aXRoaW4gdGhlIHRyYWluKCkgZnVuY3Rpb24uICBXaXRoIHRoYXQgaW4gbWluZCwgaGVyZSBpdCBpcyBvcGVyYXRpbmcKd2l0aCB0aGUga25uIG1ldGhvZC4KCiMjIyBDViB3aXRoIGtubgoKV2hlbiB0cmFpbigpIGlzIGNhbGxlZCB3aXRoIHRoZSB0ckNvbnRyb2wgYW5kIHR1bmVHcmlkLCB3ZSBjYW4gY29udHJvbApob3cgdGhlIGtubiB0cmFpbmluZyBpcyByZXBlYXRlZCwgaW4gdGhpcyBjYXNlIGl0IHdpbGwgaXRlcmF0ZSBvdmVyIGsKZnJvbSAxIHRvIDEwLgoKVGhpcyBjdXJyZW50bHkgZmFpbHMgZHVlIHRvIGEgc3RhY2sgb3ZlcmZsb3cuLi4KCmBgYHtyLCBldmFsPUZBTFNFfQpjdl9jb250cm9sIDwtIHRyYWluQ29udHJvbChtZXRob2QgPSAiY3YiLCBudW1iZXIgPSAxMCkKCmtubl90cmFpbl9maXQgPC0gdHJhaW4ob3V0Y29tZV9mYWN0b3IgfiAuLCBkYXRhID0gdHJhaW5fZGYsCiAgICAgICAgICAgICAgICAgICAgICAgbWV0aG9kID0gImtubiIsCiAgICAgICAgICAgICAgICAgICAgICAgdHJDb250cm9sID0gY3ZfY29udHJvbCwKICAgICAgICAgICAgICAgICAgICAgICB0dW5lR3JpZCA9IGRhdGEuZnJhbWUoayA9IDE6MTApKQprbm5fdHJhaW5fZml0W1siYmVzdFR1bmUiXV0KCnBsb3QoeCA9IDE6MTAsIDEgLSBrbm5fdHJhaW5fZml0JHJlc3VsdHNbLCAyXSwgcGNoID0gMTksCiAgICAgeWxhYiA9ICJwcmVkaWN0aW9uIGVycm9yIiwgeGxhYiA9ICJrIikKbGluZXMobG9lc3Muc21vb3RoKHggPSAxOjEwLCAxIC0ga25uX3RyYWluX2ZpdCRyZXN1bHRzWywgMl0sZGVncmVlID0gMiksCiAgICAgIGNvbCA9ICIjQ0MwMDAwIikKYGBgCgojIyMgQm9vdHN0cmFwIHdpdGgga25uCgpgYGB7cn0KYm9vdF9jb250cm9sIDwtIHRyYWluQ29udHJvbChtZXRob2QgPSAiYm9vdCIsIG51bWJlciA9IDIwLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgIHJldHVyblJlc2FtcCA9ICJhbGwiKQoKa25uX3RyYWluX2ZpdCA8LSB0cmFpbihvdXRjb21lIH4gLiwgZGF0YSA9IHRyYWluX2FsbCwKICAgICAgICAgICAgICAgICAgICAgICBtZXRob2QgPSAia25uIiwKICAgICAgICAgICAgICAgICAgICAgICB0ckNvbnRyb2wgPSBib290X2NvbnRyb2wsCiAgICAgICAgICAgICAgICAgICAgICAgdHVuZUdyaWQgPSBkYXRhLmZyYW1lKGsgPSAxOjEwKSkKa25uX3RyYWluX2ZpdFtbImJlc3RUdW5lIl1dCgpwbG90KHggPSAxOjEwLCAxIC0ga25uX3RyYWluX2ZpdCRyZXN1bHRzWywgMl0sIHBjaCA9IDE5LAogICAgIHlsYWIgPSAicHJlZGljdGlvbiBlcnJvciIsIHhsYWIgPSAiayIpCmxpbmVzKGxvZXNzLnNtb290aCh4ID0gMToxMCwgMSAtIGtubl90cmFpbl9maXQkcmVzdWx0c1ssIDJdLGRlZ3JlZSA9IDIpLAogICAgICBjb2wgPSAiI0NDMDAwMCIpCmBgYAoKIyMjIEV4cGxhaW4gdGhlIGltcG9ydGFudCB2YXJpYWJsZXMKCkluIHRoaXMgaW5zdGFuY2Ugd2Ugd2lsbCBzZWFyY2ggZm9yIGdlbmVzIHdoaWNoIHdlcmUgaW1wb3J0YW50IGZvciB0aGUKbW9kZWwncyBjcmVhdGlvbi4KClRoZSBEQUxFWCBwYWNrYWdlIHByb3ZpZGVzIGEgZnVuY3Rpb246IGZlYXR1cmVfaW1wb3J0YW5jZSgpIHdoaWNoCnNlZWtzIHRvIHVzZSBhIHNlcmllcyBvZiBvdGhlciBtZXRob2RzIHRvIGV4dHJhY3QgKGluIHRoaXMgY2FzZSwKZ2VuZXMpIGZlYXR1cmVzIHdoaWNoIGRvIGEgZ29vZCBqb2Igb2YgZXhwbGFpbmluZyB0aGUgcmVzdWx0IHByb2R1Y2VkCmJ5IHRoZSBtb2RlbC4gIEluIHRoZSBjYXNlIG9mIHRoaXMgZGF0YXNldCwgd2hpY2ggaGFzIHRob3VzYW5kcyBvZgpmZWF0dXJlcywgdGhpcyBkb2VzIG5vdCBhcHBlYXIgdG8gZW5kIHdlbGwuCgpgYGB7ciwgZXZhbD1GQUxTRX0KZXhwbGFpbmVyX2tubiA8LSBEQUxFWDo6ZXhwbGFpbihrbm5fZml0LCBsYWJlbCA9ICJrbm4iLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGRhdGEgPSB0cmFpbl9kZiwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB5ID0gYXMubnVtZXJpYyh0cmFpbl9vdXRjb21lcykpCgojIyBBRkFJQ1QgdGhlIGZvbGxvd2luZyB3aWxsIHRha2UgZm9yZXZlciB1bmxlc3Mgd2UgZHJhc3RpY2FsbHkgcmVkdWNlIHRoZSBjb21wbGV4aXR5IG9mIHRoZSBtb2RlbC4KIyMgeWVhaCwgSSBsZXQgaXQgcnVuIGZvciBhIHdlZWsuCiMjIGZlYXR1cmVzIDwtIGZlYXR1cmVfaW1wb3J0YW5jZShleHBsYWluZXJfa25uLCBuX3NhbXBsZSA9IDUwLCB0eXBlID0gImRpZmZlcmVuY2UiKQpgYGAKCkNvbnZlcnNlbHksIHdlIGNhbiB1c2UgdGhlIHZhckltcCgpIGZ1bmN0aW9uLi4uCgpgYGB7cn0Ka25uX3ZhcmlhYmxlX2ltcG9ydGFuY2UgPC0gdmFySW1wKGtubl90cmFpbl9maXQpCnBsb3Qoa25uX3ZhcmlhYmxlX2ltcG9ydGFuY2UsIHRvcCA9IDE1KQprbm5fdmFyaWFibGVzIDwtIGtubl92YXJpYWJsZV9pbXBvcnRhbmNlW1siaW1wb3J0YW5jZSJdXSAlPiUKICBhcnJhbmdlKGRlc2MoY3VyZSkpCmBgYAoKR2l2ZW4gdGhhdCwgd2UgY2FuIGFzayBmb3IgdGhlIHNpbWlsYXJpdHkgdG8gdGhlIERFU2VxIHJlc3VsdHMuLi4KCmBgYHtyfQprbm5fY29tcGFyaXNvbiA8LSBjb21wYXJlX21sX2RlKGtubl92YXJpYWJsZV9pbXBvcnRhbmNlLCBkZXNlcV9kZV9jZiwgYW5ub3RhdGlvbnMsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgbiA9IGNvbXBhcmlzb25fbiwgcGxvdF9uID0gMTUsIGNvbHVtbiA9ICJjdXJlIikKa25uX2NvbXBhcmlzb25bWyJjb21wYXJpc29uX3Bsb3QiXV0KY29tcGFyaXNvbl9sZmNbWyJrbm4iXV0gPC0ga25uX2NvbXBhcmlzb25bWyJzaGFyZWQiXV0KYGBgCgojIyBSYW5kb20gRm9yZXN0CgpJIHRoaW5rIFIvY2FyZXQgcHJvdmlkZXMgYSBmZXcgaW1wbGVtZW50YXRpb24gb2YgdGhlIGZvcmVzdCBmYW1pbHkgb2YKbWV0aG9kcywgdGhpcyBpcyB1c2luZyByYW5nZXIoQHdyaWdodFJhbmdlckZhc3RJbXBsZW1lbnRhdGlvbjIwMTdhKS4KClRoZSBwYXJhbWV0ZXIgJ210cnknIGlzIG9mdGVuIGltcG9ydGFudCwgaWYgSSByZWFkIHRoZSB0ZXh0IGNvcnJlY3RseQppdCBjb250cm9scyBob3cgbWFueSB2YXJpYWJsZXMgdG8gc2FtcGxlIGluIGVhY2ggc3BsaXQgb2YgdGhlIHRyZWUuClRodXMgaGlnaGVyIG51bWJlcnMgc2hvdWxkIHByZXN1bWFibHkgbWFrZSBpdCBtb3JlIHNwZWNpZmljIGF0IHRoZQpyaXNrIG9mIG92ZXJmaXR0aW5nLgoKU2V0dGluZyBtaW4ubm9kZS5zaXplIHNldHMgdGhlIG1pbmltdW1lIG5vZGUgc2l6ZSBvZiB0ZXJtaW5hbCBub2RlcyBpbgplYWNoIHRyZWUuICBFYWNoIGluY3JlbWVudCB1cCBzcGVlZHMgdGhlIGFsZ29yaXRobS4KCkkgYW0gZ29pbmcgdG8gdXNlIG15IGJvb3QgY29udHJvbCB0cmFpbmVyIGZyb20gYWJvdmUgYW5kIHNlZSBob3cgaXQgZ29lcy4KCmBgYHtyfQpyZl90cmFpbl9maXQgPC0gdHJhaW4ob3V0Y29tZSB+IC4sIGRhdGEgPSB0cmFpbl9hbGwsCiAgICAgICAgICAgICAgICBtZXRob2QgPSAicmFuZ2VyIiwgdHJDb250cm9sID0gYm9vdF9jb250cm9sLAogICAgICAgICAgICAgICAgaW1wb3J0YW5jZSA9ICJwZXJtdXRhdGlvbiIsCiAgICAgICAgICAgICAgICB0dW5lR3JpZCA9IGRhdGEuZnJhbWUoCiAgICAgICAgICAgICAgICAgICAgbXRyeSA9IDIwMCwKICAgICAgICAgICAgICAgICAgICBtaW4ubm9kZS5zaXplID0gMSwKICAgICAgICAgICAgICAgICAgICBzcGxpdHJ1bGUgPSAiZ2luaSIpLAogICAgICAgICAgICAgICAgdmVyYm9zZSA9IFRSVUUpCnJmX3RyYWluX2ZpdFtbImZpbmFsTW9kZWwiXV1bWyJwcmVkaWN0aW9uLmVycm9yIl1dCgpyZl92YXJpYWJsZV9pbXBvcnRhbmNlIDwtIHZhckltcChyZl90cmFpbl9maXQpCnBsb3QocmZfdmFyaWFibGVfaW1wb3J0YW5jZSwgdG9wID0gMTUpCnJmX3ZhcmlhYmxlcyA8LSByZl92YXJpYWJsZV9pbXBvcnRhbmNlW1siaW1wb3J0YW5jZSJdXSAlPiUKICBhcnJhbmdlKGRlc2MoT3ZlcmFsbCkpCgpyZl9wcmVkaWN0X3RyYWluZWQgPC0gcHJlZGljdChyZl90cmFpbl9maXQsIHRyYWluX2RmKQpyZl9wcmVkaWN0X2V2YWx1YXRlZCA8LSBzZWxmX2V2YWx1YXRlX21vZGVsKHJmX3ByZWRpY3RfdHJhaW5lZCwgZGF0YXNldHMsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgd2hpY2ggPSBzcGxpdCwgdHlwZSA9ICJ0cmFpbiIpCnJmX3ByZWRpY3RfZXZhbHVhdGVkCmBgYAoKIyMjIFdoYXQgd291bGQgYmUgc2hhcmVkIGJldHdlZW4gREVTZXEyIGFuZCB0aGUgTUwgY2xhc3NpZmllcj8KCmBgYHtyfQpyZl9jb21wYXJpc29uIDwtIGNvbXBhcmVfbWxfZGUocmZfdmFyaWFibGVfaW1wb3J0YW5jZSwgZGVzZXFfZGVfY2YsIGFubm90YXRpb25zLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgbiA9IGNvbXBhcmlzb25fbiwgcGxvdF9uID0gMTUpCnJmX2NvbXBhcmlzb25bWyJjb21wYXJpc29uX3Zlbm4iXV0KY29tcGFyaXNvbl9sZmNbWyJyZiJdXSA8LSByZl9jb21wYXJpc29uW1sic2hhcmVkIl1dCmBgYAoKIyMjIENvbXBhcmUgdG8gdGhlIFdHQ05BIHJlc3VsdHMKCkEgY291cGxlIG1vbnRocyBhZ28gSSBzcGVudCBhIGxpdHRsZSB0aW1lIGF0dGVtcHRpbmcgdG8gcmVjYXBpdHVsYXRlCkFsZWphbmRybydzIFdHQ05BIHJlc3VsdHMuICBJIHRoaW5rIEkgZGlkIHNvIGJ5IG1vc3RseSBjb3B5L3Bhc3RpbmcKaGlzIHdvcmsgYW5kIGFkZGluZyBzb21lIGNvbW1lbnRhcnkgYW5kIHR3ZWFraW5nIHBhcnRzIG9mIGl0IHNvIHRoYXQKaXQgd2FzIGVhc2llciBmb3IgbWUgdG8gcmVhZC91bmRlcnN0YW5kLiAgSW4gdGhlIHByb2Nlc3MsIEkgZ2VuZXJhdGVkCmEgc2VyaWVzIG9mIG1vZHVsZXMgd2hpY2ggbG9va2VkIHNpbWlsYXIvaWRlbnRpY2FsIHRvIGhpcy4KVW5mb3J0dW5hdGVseSwgSSBkaWQgbm90IGFkZCBzb21lIHNlY3Rpb25zIHRvIHJlY29yZCB0aGUgZ2VuZXMvbW9kdWxlcwp0byBzb21lIG91dHB1dCBmaWxlcy4gIEkgYW0gdGhlcmVmb3JlIGdvaW5nIGJhY2sgdG8gdGhhdCBub3cgYW5kIGRvaW5nCnNvIGluIHRoZSBob3BlcyB0aGF0IEkgY2FuIGNvbXBhcmUgdGhvc2UgbW9kdWxlcyB0byB0aGUgcmVzdWx0cwpwcm9kdWNlZCBieSB0aGUgY2xhc2lmaWVycy4KCkl0IHNlZW1zIHRoYXQgdGhlIGludGVyZXN0IGluIHRoaXMgY29tcGFyaXNvbiBoYXMgd2FuZWQsIHNvIEkgYW0ganVzdApnb2luZyB0byBkaXNhYmxlIGl0LgoKYGBge3IsIGV2YWw9RkFMU0V9CndnY25hX3Jlc3VsdCA8LSBvcGVueGxzeDo6cmVhZFdvcmtib29rKGdsdWUoImV4Y2VsL3dnY25hX2ludGVyZXN0aW5nX2dlbmVzLXZ7dmVyfS54bHN4IikpCnJvd25hbWVzKHdnY25hX3Jlc3VsdCkgPC0gd2djbmFfcmVzdWx0W1sicm93Lm5hbWVzIl1dCndnY25hX3Jlc3VsdFtbInJvdy5uYW1lcyJdXSA8LSBOVUxMCgp0b3BfbWwgPC0gcm93bmFtZXMoaGVhZChyZl92YXJpYWJsZXMsIG4gPSBjb21wYXJpc29uX24pKQpjb21wYXJpc29uIDwtIGxpc3QoIndnY25hIiA9IHJvd25hbWVzKHdnY25hX3Jlc3VsdCksICJtbCIgPSB0b3BfbWwpCmNvbXBhcmlzb25fdmVubiA8LSBWZW5uZXJhYmxlOjpWZW5uKGNvbXBhcmlzb24pCmNvbXBhcmlzb25fd2djbmFbWyJyZiJdXSA8LSBhbm5vdGF0aW9uc1tjb21wYXJpc29uX3Zlbm5ASW50ZXJzZWN0aW9uU2V0c1siMTEiXVtbMV1dLCBdClZlbm5lcmFibGU6OnBsb3QoY29tcGFyaXNvbl92ZW5uLCBkb1dlaWdodHMgPSBGQUxTRSwgdHlwZSA9ICJjaXJjbGVzIikKYGBgCgojIyMjIERpZ3Jlc3Npb24gZG8gdGhlIGdlbmVzIHByb3ZpZGVkIGJ5IHZhckltcCBtZWFuIGFueXRoaW5nPwoKTGV0IHVzIHRha2UgYSBtb21lbnQgYW5kIHNlZSBpZiB0aGUgdG9wLW4gZ2VuZXMgcmV0dXJuZWQgYnkgdmFySW1wKCkKaGF2ZSBzb21lIG1lYW5pbmcgd2hpY2gganVtcHMgb3V0LiAgT25lIG1pZ2h0IGFzc3VtZSwgZ2l2ZW4gb3VyIGV4dGFudApEaWZmZXJlbnRpYWwgRXhwcmVzc2lvbiByZXN1bHRzLCB0aGF0IHRoZSBpbnRlcmxldWtpbiByZXNwb25zZSB3aWxsIGJlCmEgbGlrZWx5IGNhbmRpZGF0ZS4KCmBgYHtyfQppbXBvcnRhbmNlX2dwIDwtIHNpbXBsZV9ncHJvZmlsZXIocm93bmFtZXMoaGVhZChyZl92YXJpYWJsZXMsIG4gPSBjb21wYXJpc29uX24pKSkKaW1wb3J0YW5jZV9ncApgYGAKCiMjIyBOb3cgdGhlIHJhbmRvbSBmb3Jlc3QgdGVzdGVycyEKCmBgYHtyfQpyZl9wcmVkaWN0X3Rlc3QgPC0gcHJlZGljdChyZl90cmFpbl9maXQsIHRlc3RfZGYpCgpyZl9wcmVkaWN0X3Rlc3RfZXZhbHVhdGVkIDwtIHNlbGZfZXZhbHVhdGVfbW9kZWwocmZfcHJlZGljdF90ZXN0LCBkYXRhc2V0cywKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHdoaWNoID0gc3BsaXQsIHR5cGUgPSAidGVzdCIpCnJmX3ByZWRpY3RfdGVzdF9ldmFsdWF0ZWQKYGBgCgojIyBHTE0sIG9yIExvZ2lzdGljIHJlZ3Jlc3Npb24gYW5kIHJlZ3VsYXJpemF0aW9uCgpMb2dpc3RpYyByZWdyZXNzaW9uIGlzIGEgc3RhdGlzdGljYWwgbWV0aG9kIGZvciBiaW5hcnkgcmVzcG9uc2VzLgpIb3dldmVyLCBpdCBpcyBhYmxlIHRvIHdvcmsgd2l0aCBtdWx0aXBsZSBjbGFzc2VzIGFzIHdlbGwuICBUaGUKZ2VuZXJhbCBpZGVhIG9mIHRoaXMgbWV0aG9kIGlzIHRvIGZpbmQgcGFyYW1ldGVycyB3aGljaCBpbmNyZWFzZSB0aGUKbGlrZWxpaG9vZCB0aGF0IHRoZSBvYnNlcnZlZCBkYXRhIGlzIHNhbXBsZWQgZnJvbSBhIHN0YXRpc3RpY2FsCmRpc3RyaWJ1dGlvbiBvZiBpbnRlcmVzdC4gIFRoZSB0cmFuc2Zvcm1hdGlvbnMgYW5kIGxpbmVhcgpyZWdyZXNzaW9uLWVzcXVlIHRhc2tzIHBlcmZvcm1lZCBhcmUgY29uZnVzaW5nLCBidXQgb25jZSB0aG9zZSBhcmUKcGVyZm9ybWVkLCB0aGUgdGFzayBiZWNvbWVzIHNldHRpbmcgdGhlIG1vZGVsJ3MgKGZpdHRpbmcpIHBhcmFtZXRlcnMKdG8gdmFsdWVzIHdoaWNoIGluY3JlYXNlIHRoZSBwcm9iYWJpbGl0eSB0aGF0IHRoZSBzdGF0aXN0aWNhbCBtb2RlbApsb29rcyBsaWtlIHRoZSBhY3R1YWwgZGF0YXNldCBnaXZlbiB0aGUgdHJhaW5pbmcgZGF0YSwgYW5kIHRoYXQgd2hlbgpzYW1wbGVzLCB3aWxsIHJldHVybiB2YWx1ZXMgd2hpY2ggYXJlIHNpbWlsYXIuICBUaGUgbW9zdCBsaWtlbHkKc3RhdGlzdGljYWwgZGlzdHJpYnV0aW9ucyBvbmUgd2lsbCB3YW50IHRvIGZpdCBhcmUgdGhlIEdhdXNzaWFuLCBpbgp3aGljaCBjYXNlIHdlIHdhbnQgdG8gdHJhbnNmb3JtL25vcm1hbGl6ZSB0aGUgbWVhbi92YXJpYW5jZSBvZiBvdXIKdmFyaWFibGVzIHNvIHRoZXkgbG9vayB3aGF0ZXZlciBub3JtYWwgZGlzdHJpYnV0aW9uIHdlIGFyZSB1c2luZy4KQ29udmVyc2VseSwgbG9naXN0aWMgcmVncmVzc2lvbiB1c2VzIGEgYmlubm9taWFsIGRpc3RyaWJ1dGlvbiAobGlrZQpvdXIgcmF3IHNlcXVlbmNpbmcgZGF0YSEpIGJ1dCB3aGljaCBpcyBmcm9tIDAtMS4KCiMjIyBVc2luZyBhIHNpbmdsZSBnZW5lCgpMZXQgdXMgdGFrZSB0aGUgbW9zdCBpbXBvcnRhbnQgZ2VuZSBvYnNlcnZlZCBpbiBvbmUgb2Ygb3VyIHByZXZpb3VzCnRyYWluaW5nIHNldHM6IEVOU0cwMDAwMDI0ODQwNSBQUlI1LUFSSEdBUDgKCmBgYHtyLCBldmFsPUZBTFNFfQpnZW5lX2lkIDwtICJFTlNHMDAwMDAyNDg0MDUiCnNpbmdsZV9maXQgPC0gdHJhaW4oCiAgICBvdXRjb21lIH4gRU5TRzAwMDAwMjQ4NDA1LCBkYXRhID0gdHJhaW5fYWxsLAogICAgbWV0aG9kID0gImdsbSIsIGZhbWlseSA9ICJiaW5vbWlhbCIsIHRyQ29udHJvbCA9IHRyYWluQ29udHJvbCgibm9uZSIpKQoKdHQgPC0gZGF0YS5mcmFtZSgiRU5TRzAwMDAwMjQ4NDA1IiA9IHNlcShtaW4odHJhaW5fZGZbW2dlbmVfaWRdXSksCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgbWF4KHRyYWluX2RmW1tnZW5lX2lkXV0pLCBsZW4gPSAxMDApKQojIyBwcmVkaWN0IHByb2JhYmlsaXRpZXMgZm9yIHRoZSBzaW11bGF0ZWQgZGF0YQp0dCRzdWJ0eXBlID0gcHJlZGljdChzaW5nbGVfZml0LCBuZXdkYXRhID0gdHQsIHR5cGU9InByb2IiKVssIDFdCiMjIHBsb3QgdGhlIHNpZ21vaWQgY3VydmUgYW5kIHRoZSB0cmFpbmluZyBkYXRhCnBsb3QoaWZlbHNlKG91dGNvbWUgPT0gImN1cmUiLCAxLCAwKSB+IEVOU0cwMDAwMDI0ODQwNSwKICAgICBkYXRhID0gdHJhaW5fYWxsLCBjb2wgPSAicmVkNCIsCiAgICAgeWxhYiA9ICJDRiBhcyAwIG9yIDEiLCB4bGFiID0gImZhdm9yaXRlIGdlbmUgZXhwcmVzc2lvbiIpCmxpbmVzKHN1YnR5cGUgfiBFTlNHMDAwMDAyNDg0MDUsIHR0LCBjb2wgPSAiZ3JlZW40IiwgbHdkID0gMikKCnBsb3RfZGYgPC0gdHJhaW5fYWxsWywgYygib3V0Y29tZSIsICJFTlNHMDAwMDAyNDg0MDUiKV0KZ2diZXR3ZWVuc3RhdHMocGxvdF9kZiwgIm91dGNvbWUiLCAiRU5TRzAwMDAwMjQ4NDA1IikKYGBgCgpIYXZpbmcgdHJpZWQgd2l0aCAxIGdlbmUsIGxldCB1cyBleHRlbmQgdGhpcyB0byBhbGwgZ2VuZXMuICBJbiBteQpmaXJzdCB0cnkgb2YgdGhpcywgaXQgdG9vayBhIGxvbmcgdGltZS4KCmBgYHtyfQpnbG1fdHJhaW5fZml0IDwtIHRyYWluKG91dGNvbWUgfiAuLCBkYXRhID0gdHJhaW5fYWxsLAogICAgICAgICAgICAgICAgIHRyQ29udHJvbCA9IGJvb3RfY29udHJvbCwKICAgICAgICAgICAgICAgICBtZXRob2QgPSAiZ2xtIiwgZmFtaWx5ID0gImJpbm9taWFsIikKYGBgCgojIyBDb21wYXJlIEdMTSBhbmQgV0dDTkEvREUKClRoZSBwcmV2aW91cyBibG9jayBkb2VzIG5vdCB0YWtlIGludG8gYWNjb3VudCBvdXIgcGFyYW1ldGVyIHN3ZWVwLCBzbwppdCBpc24ndCB0aGVvcmV0aWNhbGx5IHZlcnkgdXNlZnVsIGZvciB0aGlzIHNlcmllcyBvZiBtZXRob2xvZ2ljYWwKdGVzdHMuICBBcyBhIHJlc3VsdCwgdGhlIGZvbGxvd2luZyBibG9jayBpcyBhbHNvIG5vdCBwYXJ0aWN1bGFybHkKaGVscGZ1bCAtLSB0aG91Z2ggSSBzdXNwZWN0IHRoZSByZXN1bHRzIGFyZSBpZGVudGljYWwgdG8gd2hhdCBmb2xsb3dzLgoKYGBge3J9CmdsbV92YXJpYWJsZV9pbXBvcnRhbmNlIDwtIHZhckltcChnbG1fdHJhaW5fZml0KQojIyBPaCwgdGhpcyBvbmx5IHByb2R1Y2VzIDEwMCBlbnRyaWVzIC0tIHNvIG1lIGdldHRpbmcgdGhlIHRvcCA0MDAgaXMgc2lsbHkuCmdsbV92YXJpYWJsZXMgPC0gZ2xtX3ZhcmlhYmxlX2ltcG9ydGFuY2VbWyJpbXBvcnRhbmNlIl1dICU+JQogIGFycmFuZ2UoZGVzYyhPdmVyYWxsKSkKdHQgPSBwbG90KGdsbV92YXJpYWJsZV9pbXBvcnRhbmNlLCB0b3AgPSAxNSkKc2ltcGxlX2dwcm9maWxlcihyb3duYW1lcyhoZWFkKGdsbV92YXJpYWJsZXMsIG4gPSBjb21wYXJpc29uX24pKSkKCnRvcF9nbG0gPC0gcm93bmFtZXMoZ2xtX3ZhcmlhYmxlcykKY29tcGFyaXNvbiA8LSBsaXN0KCJkZSIgPSB0b3BfYm90dG9tX2lkcywgIm1sIiA9IHRvcF9nbG0pCmNvbXBhcmlzb25fdmVubiA8LSBWZW5uZXJhYmxlOjpWZW5uKGNvbXBhcmlzb24pCmNvbXBhcmlzb25fbGZjW1siZ2xtIl1dIDwtIGZEYXRhKGNfbW9ub2N5dGVzKVtjb21wYXJpc29uX3Zlbm5ASW50ZXJzZWN0aW9uU2V0c1siMTEiXVtbMV1dLCBdClZlbm5lcmFibGU6OnBsb3QoY29tcGFyaXNvbl92ZW5uLCBkb1dlaWdodHMgPSBGQUxTRSwgdHlwZSA9ICJjaXJjbGVzIikKYGBgCgpEcm9wcGluZyB0aGUgYWRqcCBhbmQgd2djbmEgY29tcGFyaXNvbnMsIHRoZXkgc2hvdyBsaXR0bGUgb3ZlcmxhcC4KCmBgYHtyLCBldmFsPUZBTFNFfQpjb21wYXJpc29uIDwtIGxpc3QoImRlIiA9IGxvd2VzdF9hZGpwLCAibWwiID0gdG9wX2dsbSkKY29tcGFyaXNvbl92ZW5uIDwtIFZlbm5lcmFibGU6OlZlbm4oY29tcGFyaXNvbikKY29tcGFyaXNvbl9hZGpwW1siZ2xtIl1dIDwtIGZEYXRhKGNfbW9ub2N5dGVzKVtjb21wYXJpc29uX3Zlbm5ASW50ZXJzZWN0aW9uU2V0c1siMTEiXVtbMV1dLCBdClZlbm5lcmFibGU6OnBsb3QoY29tcGFyaXNvbl92ZW5uLCBkb1dlaWdodHMgPSBGQUxTRSwgdHlwZSA9ICJjaXJjbGVzIikKCmNvbXBhcmlzb24gPC0gbGlzdCgiZGUiID0gaGlnaGVzdF9leHBycywgIm1sIiA9IHRvcF9nbG0pCmNvbXBhcmlzb25fdmVubiA8LSBWZW5uZXJhYmxlOjpWZW5uKGNvbXBhcmlzb24pClZlbm5lcmFibGU6OnBsb3QoY29tcGFyaXNvbl92ZW5uLCBkb1dlaWdodHMgPSBGQUxTRSwgdHlwZSA9ICJjaXJjbGVzIikKYGBgCgpJbiB0aGlzIGJsb2NrLCBJIHJlcGVhdCB0aGUgYWJvdmUgd2l0aCB0aGUgdHVuZUdyaWQgb3B0aW9uIHNldCBzbyB0aGF0Cml0IG1heSBkbyB0aGUgdHJhaW5pbmcgdXNpbmcgYSBzd2VlcCBvZiBwdXRhdGl2ZWx5IGhlbHBmdWwgcGFyYW1ldGVyczsKYXMgYSByZXN1bHQgaXQgc2hvdWxkIHByb3ZpZGUgYSBtb3JlIGFwcHJvcHJpYXRlIChidXQgcXVpdGUgcG9zc2libHkKaWRlbnRpY2FsKSB0cmFpbmVyLgoKYGBge3J9CiMjcmZfbWV0aG9kIDwtIHRyYWluQ29udHJvbChtZXRob2QgPSAicmFuZ2VyIiwgbnVtYmVyID0gMTAsIHZlcmJvc2UgPSBUUlVFKQojIyB0cmFpbl9tZXRob2QgPC0gdHJhaW5Db250cm9sKG1ldGhvZCA9ICJjdiIsIG51bWJlciA9IDEwKQpnbG1fZml0IDwtIHRyYWluKG91dGNvbWUgfiAuLCBkYXRhID0gdHJhaW5fYWxsLCBtZXRob2QgPSAiZ2xtbmV0IiwKICAgICAgICAgICAgICAgICB0ckNvbnRyb2wgPSBib290X2NvbnRyb2wsIGltcG9ydGFuY2UgPSAicGVybXV0YXRpb24iLAogICAgICAgICAgICAgICAgIHR1bmVHcmlkID0gZGF0YS5mcmFtZSgKICAgICAgICAgICAgICAgICAgIGFscGhhID0gMC41LAogICAgICAgICAgICAgICAgICAgbGFtYmRhID0gc2VxKDAuMSwgMC43LCAwLjA1KSksCiAgICAgICAgICAgICAgICAgdmVyYm9zZSA9IFRSVUUpCmdsbV9maXQKCmdsbV9wcmVkaWN0X3RyYWluZWQgPC0gcHJlZGljdChnbG1fZml0LCB0cmFpbl9kZikKCmdsbV90cmFpbl9ldmFsIDwtIHNlbGZfZXZhbHVhdGVfbW9kZWwoZ2xtX3ByZWRpY3RfdHJhaW5lZCwgZGF0YXNldHMsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgd2hpY2ggPSBzcGxpdCwgdHlwZSA9ICJ0cmFpbiIpCmdsbV90cmFpbl9ldmFsCmBgYAoKIyMjIE5vdyB0aGUgR0xNIHRlc3RlcnMhCgpgYGB7cn0KZ2xtX3ByZWRpY3RfdGVzdCA8LSBwcmVkaWN0KGdsbV9maXQsIHRlc3RfZGYpCgpnbG1fZml0X2V2YWxfdGVzdCA8LSBzZWxmX2V2YWx1YXRlX21vZGVsKGdsbV9wcmVkaWN0X3Rlc3QsIGRhdGFzZXRzLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHdoaWNoID0gc3BsaXQsIHR5cGUgPSAidGVzdCIpCmdsbV9maXRfZXZhbF90ZXN0CmBgYAoKIyMjIENvbXBhcmUgYWdhaW4gdnMgdGhlIERFIHRhYmxlCgpJbiB0aGUgZm9sbG93aW5nIGJsb2NrIEkgZXh0cmFjdCB0aGUgdG9wIGFuZCBib3R0b20gREUgZ2VuZXMgdnMuIHRoZQp0b3Agc2V0IG9mIGdlbmVzIGluIHRoZSBnbG1fdmFyaWFibGVzIG9yZGVyZWQgYnkgdGhlICdPdmVyYWxsJyBjb2x1bW4uCgpJIHNob3VsZCBtYWtlIHRoaXMgYSBmdW5jdGlvbiwgdGhlcmUgaXMgbm8gd2F5IHRoaXMgd2lsbCBrZXB0IGNvcnJlY3QKYWNyb3NzIG11bHRpcGxlIGl0ZXJhdGlvbnMgb2YgdGhlIG1sL3dnY25hL2V0YyBjb21wYXJpc29ucy4KCkkgYW0gZ29pbmcgdG8gc3RvcCB0cnlpbmcgdG8gY29tcGFyZSB0aGVzZSByZXN1bHRzIGFjcm9zcwpXR0NOQS9hZGpwL2V4cHJlc3Npb24sIHRob3NlIGNvbXBhcmlzb25zIGFyZSBhbGwgYmFzaWNhbGx5IG51bGwuCgpgYGB7cn0KZ2xtX3ZhcmlhYmxlX2ltcG9ydGFuY2UgPC0gdmFySW1wKGdsbV9maXQpCmdsbV9jb21wYXJpc29uIDwtIGNvbXBhcmVfbWxfZGUoZ2xtX3ZhcmlhYmxlX2ltcG9ydGFuY2UsIGRlc2VxX2RlX2NmLCBhbm5vdGF0aW9ucywKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBuID0gY29tcGFyaXNvbl9uLCBwbG90X24gPSAxNSkKY29tcGFyaXNvbl9sZmNbWyJnbG0iXV0gPC0gZ2xtX2NvbXBhcmlzb25bWyJzaGFyZWQiXV0KYGBgCgojIyBHcmFkaWVudCBCb29zdGVyCgpBdCB0aGlzIHBvaW50IEkgYmFzaWNhbGx5IGhhdmUgYSBmZWVsaW5nIGZvciBob3cgb25lIG1heSB1c2UgdGhlCnR1bmVHcmlkIHRvIGFwcGx5IGEgcGFyYW1ldGVyIHN3ZWVwIG9mIHRoZSBkYXRhIGFsb25nIHdpdGggdGhlIHZhcmlvdXMKKGluIHRoaXMgY2FzZSBjdikgc2FtcGxpbmcgbWV0aG9kcy4gIFRodXMsIHRoZSBmb2xsb3dpbmcgaXMgYmFzaWNhbGx5CmlkZW50aWNhbCB0byB0aGUgcHJldmlvdXMgYmxvY2tzIGV4Y2VwdCBpdCBpcyB1c2luZyB4Z2JUcmVlICh3aGljaCBpcwp0aGUgUiBpbXBsZW1lbnRhdGlvbiBvZiBleHRyZW1lIGJvb3N0aW5nOgooQGNoZW5YR0Jvb3N0U2NhbGFibGVUcmVlMjAxNmEpKS4KCmBgYHtyfQojI3JmX21ldGhvZCA8LSB0cmFpbkNvbnRyb2wobWV0aG9kID0gInJhbmdlciIsIG51bWJlciA9IDEwLCB2ZXJib3NlID0gVFJVRSkKdHJhaW5fbWV0aG9kIDwtIHRyYWluQ29udHJvbChtZXRob2QgPSAiY3YiLCBudW1iZXIgPSAxMCkKCmdiX2ZpdCA8LSB0cmFpbihvdXRjb21lIH4gLiwgZGF0YSA9IHRyYWluX2FsbCwKICAgICAgICAgICAgICAgIG1ldGhvZCA9ICJ4Z2JUcmVlIiwgdHJDb250cm9sID0gdHJhaW5fbWV0aG9kLAogICAgICAgICAgICAgICAgdHVuZUdyaWQgPSBkYXRhLmZyYW1lKAogICAgICAgICAgICAgICAgICAgIG5yb3VuZHMgPSAyMDAsCiAgICAgICAgICAgICAgICAgICAgZXRhID0gYygwLjA1LCAwLjEsIDAuMyksCiAgICAgICAgICAgICAgICAgICAgbWF4X2RlcHRoID0gNCwKICAgICAgICAgICAgICAgICAgICBnYW1tYSA9IDAsCiAgICAgICAgICAgICAgICAgICAgY29sc2FtcGxlX2J5dHJlZSA9IDEsCiAgICAgICAgICAgICAgICAgICAgc3Vic2FtcGxlID0gMC41LAogICAgICAgICAgICAgICAgICAgIG1pbl9jaGlsZF93ZWlnaHQgPSAxKSwKICAgICAgICAgICAgICAgIHZlcmJvc2UgPSBUUlVFKQoKZ2JfcHJlZGljdF90cmFpbmVkIDwtIHByZWRpY3QoZ2JfZml0LCB0cmFpbl9kZikKZ2JfcHJlZGljdF90cmFpbmVkCgpnYl90cmFpbl9ldmFsIDwtIHNlbGZfZXZhbHVhdGVfbW9kZWwoZ2JfcHJlZGljdF90cmFpbmVkLCBkYXRhc2V0cywKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHdoaWNoID0gc3BsaXQsIHR5cGUgPSAidHJhaW4iKQpnYl90cmFpbl9ldmFsCmBgYAoKIyMjIE5vdyB0aGUgR0IgdGVzdGVycyEKCmBgYHtyfQpnYl9wcmVkaWN0X3Rlc3QgPC0gcHJlZGljdChnYl9maXQsIHRlc3RfZGYpCgpnYl9wcmVkaWN0X3Rlc3RfZXZhbHVhdGVkIDwtIHNlbGZfZXZhbHVhdGVfbW9kZWwoZ2JfcHJlZGljdF90ZXN0LCBkYXRhc2V0cywKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHdoaWNoID0gc3BsaXQsIHR5cGUgPSAidGVzdCIpCmdiX3ByZWRpY3RfdGVzdF9ldmFsdWF0ZWQKYGBgCgpgYGB7cn0KZ2JfdmFyaWFibGVfaW1wb3J0YW5jZSA8LSB2YXJJbXAoZ2JfZml0KQpnYl92YXJpYWJsZXMgPC0gZ2xtX3ZhcmlhYmxlX2ltcG9ydGFuY2VbWyJpbXBvcnRhbmNlIl1dICU+JQogIGFycmFuZ2UoZGVzYyhPdmVyYWxsKSkKcGxvdChnYl92YXJpYWJsZV9pbXBvcnRhbmNlLCB0b3AgPSAxNSkKZ2JfY29tcGFyaXNvbiA8LSBjb21wYXJlX21sX2RlKGdiX3ZhcmlhYmxlX2ltcG9ydGFuY2UsIGRlc2VxX2RlX2NmLCBhbm5vdGF0aW9ucywgbiA9IGNvbXBhcmlzb25fbiwgcGxvdF9uID0gMTUpCmNvbXBhcmlzb25fbGZjW1siZ2IiXV0gPC0gZ2JfY29tcGFyaXNvbltbInNoYXJlZCJdXQpgYGAKCiMgU2hhcmVkIGltcG9ydGFuY2UKCmBgYHtyfQp1cHNldF9sZmMgPC0gbGlzdCgpCnVwc2V0X2FkanAgPC0gbGlzdCgpCnVwc2V0X3dnY25hIDwtIGxpc3QoKQpmb3IgKGQgaW4gMTpsZW5ndGgoY29tcGFyaXNvbl9sZmMpKSB7CiAgbmFtZSA8LSBuYW1lcyhjb21wYXJpc29uX2xmYylbZF0KICB1cHNldF9sZmNbW25hbWVdXSA8LSByb3duYW1lcyhjb21wYXJpc29uX2xmY1tbbmFtZV1dKQogIHVwc2V0X2FkanBbW25hbWVdXSA8LSByb3duYW1lcyhjb21wYXJpc29uX2FkanBbW25hbWVdXSkKICAjI3Vwc2V0X3dnY25hW1tuYW1lXV0gPC0gcm93bmFtZXMoY29tcGFyaXNvbl93Z2NuYVtbbmFtZV1dKQp9CgpzdGFydF9sZmMgPC0gVXBTZXRSOjpmcm9tTGlzdCh1cHNldF9sZmMpCmxmY192c19tbCA8LSBVcFNldFI6OnVwc2V0KHN0YXJ0X2xmYykKYGBgCgojIyBBIGRpZmZlcmVudCB3YXkgdG8gcXVlcnkgc2hhcmVkL3VuaXF1ZSBnZW5lcwoKVGhlIHZhcmlvdXMgZ2VuZSBzZXRzIG9ic2VydmVkIGJ5IHRoZXNlIG1ldGhvZHMgbWF5IGFsc28gYmUgY29tcGFyZWQKZGlyZWN0bHkgYnkgYW4gdXBzZXQgcGxvdCB1c2luZyB0aGUgZGVzZXFfZGVfY2Ygd2l0aAooZ2J8Z2xtfHJmfGtubilfdmFyaWFibGVfaW1wb3J0YW5jZS4uLgoKYGBge3J9CmNvbXBfbnVtYmVyIDwtIDMwMApkZV9tbF9zaWdfbGlzdCA8LSBsaXN0KAogICJERVNlcTIiID0gcm93bmFtZXMoZGVzZXFfZGVfY2YpLAogICJrbm4iID0gcm93bmFtZXMoaGVhZChrbm5fdmFyaWFibGVzLCBuID0gY29tcF9udW1iZXIpKSwKICAicmYiID0gcm93bmFtZXMoaGVhZChyZl92YXJpYWJsZXMsIG4gPSBjb21wX251bWJlcikpLAogICJnbG0iID0gcm93bmFtZXMoaGVhZChnbG1fdmFyaWFibGVzLCBuID0gY29tcF9udW1iZXIpKSwKICAiZ2IiID0gcm93bmFtZXMoaGVhZChnYl92YXJpYWJsZXMsIG4gPSBjb21wX251bWJlcikpKQppbXBvcnRhbnRfZ2VuZXNfdXBzZXRfaW5wdXQgPC0gVXBTZXRSOjpmcm9tTGlzdChkZV9tbF9zaWdfbGlzdCkKaW1wb3J0YW50X2dlbmVzX3Vwc2V0IDwtIFVwU2V0Ujo6dXBzZXQoaW1wb3J0YW50X2dlbmVzX3Vwc2V0X2lucHV0KQpwcmludChpbXBvcnRhbnRfZ2VuZXNfdXBzZXQpCgppbXBvcnRhbnRfZ2VuZXNfb3ZlcmxhcCA8LSBvdmVybGFwX2dyb3VwcyhkZV9tbF9zaWdfbGlzdCkKc2hhcmVkX2dlbmVfaWRzIDwtIG92ZXJsYXBfZ2VuZWlkcyhpbXBvcnRhbnRfZ2VuZXNfb3ZlcmxhcCwgIkRFU2VxMjprbm4iKQpzaGFyZWRfYW5ub3RhdGlvbnMgPC0gZkRhdGEoaW5wdXRfZGF0YSlbYXMuY2hhcmFjdGVyKHNoYXJlZF9nZW5lX2lkcyksIF0KCmltcG9ydGFudF9nZW5lc19vdmVybGFwIDwtIG92ZXJsYXBfZ3JvdXBzKGRlX21sX3NpZ19saXN0KQpzaGFyZWRfZ2VuZV9pZHMgPC0gb3ZlcmxhcF9nZW5laWRzKGltcG9ydGFudF9nZW5lc19vdmVybGFwLCAiREVTZXEyOnJmIikKc2hhcmVkX2Fubm90YXRpb25zIDwtIGZEYXRhKGlucHV0X2RhdGEpW2FzLmNoYXJhY3RlcihzaGFyZWRfZ2VuZV9pZHMpLCBdCgppbXBvcnRhbnRfZ2VuZXNfb3ZlcmxhcCA8LSBvdmVybGFwX2dyb3VwcyhkZV9tbF9zaWdfbGlzdCkKc2hhcmVkX2dlbmVfaWRzIDwtIG92ZXJsYXBfZ2VuZWlkcyhpbXBvcnRhbnRfZ2VuZXNfb3ZlcmxhcCwgIkRFU2VxMjpnYiIpCnNoYXJlZF9hbm5vdGF0aW9ucyA8LSBmRGF0YShpbnB1dF9kYXRhKVthcy5jaGFyYWN0ZXIoc2hhcmVkX2dlbmVfaWRzKSwgXQoKaW1wb3J0YW50X2dlbmVzX292ZXJsYXAgPC0gb3ZlcmxhcF9ncm91cHMoZGVfbWxfc2lnX2xpc3QpCnNoYXJlZF9nZW5lX2lkcyA8LSBvdmVybGFwX2dlbmVpZHMoaW1wb3J0YW50X2dlbmVzX292ZXJsYXAsICJERVNlcTI6Z2xtIikKc2hhcmVkX2Fubm90YXRpb25zIDwtIGZEYXRhKGlucHV0X2RhdGEpW2FzLmNoYXJhY3RlcihzaGFyZWRfZ2VuZV9pZHMpLCBdCmBgYAoKIyBDb2xsYXRlIHNoYXJlZCBnZW5lcyBieSBtZXRob2QgYW5kIHdyaXRlIHRoZW0KCk92ZXIgdGhlIGNvdXJzZSBvZiB0aGlzIGRvY3VtZW50IEkgcmVjb3JkZWQgdGhlIGxpbWl0ZWQgYWdyZWVtZW50CmJldHdlZW4gdGhlIERFU2VxIHJlc3VsdCBhbmQgdGhlIE1MIGNsYXNzaWZpZXIgaW4gdGhlIGRhdGEgc3RydWN0dXJlCidjb21wYXJpc29uX2xmYycgIExldHMgd3JpdGUgdGhhdCBvdXQgdG8gYW4geGxzeCBmaWxlLi4uCgpgYGB7cn0KZGVfbWxfeGxzeCA8LSBnbHVlKCJleGNlbC90Y19jb21wYXJlX2RlX21sX3RvcDNrdmFyaWFudC12e3Zlcn0ueGxzeCIpCnhsc3ggPC0gaW5pdF94bHN4KGRlX21sX3hsc3gpCndiIDwtIHhsc3hbWyJ3YiJdXQpleGNlbF9iYXNlbmFtZSA8LSB4bHN4W1siYmFzZW5hbWUiXV0Kd3JpdHRlbiA8LSB3cml0ZV94bHN4KGRhdGEgPSBjb21wYXJpc29uX2xmY1tbImtubiJdXSwgd2IgPSB3Yiwgc2hlZXQgPSAia25uX3ZzX2Rlc2VxIikKd3JpdHRlbiA8LSB3cml0ZV94bHN4KGRhdGEgPSBjb21wYXJpc29uX2xmY1tbInJmIl1dLCB3YiA9IHdiLCBzaGVldCA9ICJyZl92c19kZXNlcSIpCndyaXR0ZW4gPC0gd3JpdGVfeGxzeChkYXRhID0gY29tcGFyaXNvbl9sZmNbWyJnbG0iXV0sIHdiID0gd2IsIHNoZWV0ID0gImdsbV92c19kZXNlcSIpCndyaXR0ZW4gPC0gd3JpdGVfeGxzeChkYXRhID0gY29tcGFyaXNvbl9sZmNbWyJnYiJdXSwgd2IgPSB3Yiwgc2hlZXQgPSAiZ2JfdnNfZGVzZXEiKQpleGNlbF9yZXQgPC0gb3Blbnhsc3g6OnNhdmVXb3JrYm9vayh3YiwgZGVfbWxfeGxzeCwgb3ZlcndyaXRlID0gVFJVRSkKYGBgCgpJIHJlYWxpemVkIHNvbWV0aGluZyBpbXBvcnRhbnQgaW4gdGhpcyBibG9jazogYXQgbm8gcG9pbnQgaW4gYW55IG9mIG15CmFuYWx5c2VzIGRvIEkgcmVhbGx5IHJlY29yZCB0aGUgY2xhc3NpZmljYXRpb25zLCBJIGp1c3QgcmVjb3JkIHRoZWlyCihkaXMpY29uY29yZGFuY2Ugd2l0aCBvdXIga25vd24gYW5ub3RhdGlvbnMuICBJbiB0aGUgZm9sbG93aW5nIGJsb2NrIEkKaXRlcmF0ZSBvdmVyIG11bHRpcGxlIHJvdW5kcyBvZiB0aGUgdmFyaW91cyBjbGFzc2lmaWVycywgbGV0IHVzIG1vZGlmeQp0aGF0IGZ1bmN0aW9uIHRvIHJlY29yZCB0aGlzIGluZm9ybWF0aW9uIGluIHNvbWUgdXNlZnVsIGZhc2hpb24uCgojIFBlcmZvcm1pbmcgbXVsdGlwbGUgdHJhaW4vdGVzdCByb3VuZHMgaW4gb25lIGNhbGwKCkRvaW5nIGFsbCBvZiB0aGUgYWJvdmUgZm9yIGVhY2ggbWV0aG9kIGFuZCBjb2xsZWN0aW5nIHRoZSByZXN1bHRzIGlzCnN1cGVyIGZpZGRseSwgc28gSSB3cm90ZSBhIGxpdHRsZSBmdW5jdGlvbiB0byB0cnkgdG8gbWFrZSB0aGF0IGVhc2llci4KCldBUk5JTkc6IFR1cm5pbmcgb24gdGhlIGZvbGxvd2luZyBibG9jayB0YWtlcyB+IDIgZGF5cyBvbiBvdXIgcHJldHR5Cm5pY2Ugc2VydmVyLiAgWW91ciBtaWxlYWdlIG1heSB2YXJ5LgoKIyMgU2V0dGluZyB1cCBpbnB1dCBkYXRhIGZvciBtdWx0aXBsZSBNTCBpdGVyYXRpb25zCgpUaGVyZSBhcmUgZm91ciB2ZXJzaW9ucyBvZiB0aGUgZGF0YSB0aGF0IE1hcmlhIEFkZWxhaWRhIGFuZCBOYWppYiBhcmUKaW50ZXJlc3RlZCBpbjoKCiogVHVtYWNvK0NhbGkgYWxsIHZpc2l0cwoqIFR1bWFjbytDYWxpIHZpc2l0IDEKKiBUdW1hY28gYWxsIHZpc2l0cwoqIFR1bWFjbyB2aXNpdCAxCgpFcmdvLCBsZXQgdXMgc2V0IHVwIGFuIGFwcHJvcHJpYXRlIG5hbWluZyBjb252ZW50aW9uIGZvciB0aGVtIHRvIG1ha2UKdGhlc2UgaXRlcmF0aXZlIHJ1bnMgbW9yZSBjb2hlcmVudC4gIEkgd2lsbCBjb250aW51ZSB1c2luZyAndGMnIGFuZAondCcgZm9yIHRoZSBjbGluaWNzIGFzIGEgcHJlZml4IGFuZCBmb2xsb3cgaXQgd2l0aCAndmFsbCcgdnMuICd2MScuCgpgYGB7cn0KdGNfdmFsbF9pbnB1dCA8LSBzdWJzZXRfZXhwdCh0Y19jbGluaWNhbCwgc3Vic2V0ID0gImJhdGNoIT0nYmlvcHN5JyIpCnRfdmFsbF9pbnB1dCA8LSBzdWJzZXRfZXhwdCh0Y192YWxsX2lucHV0LCBzdWJzZXQgPSAiY2xpbmljPT0ndHVtYWNvJyIpCnRjX3YxX2lucHV0IDwtIHN1YnNldF9leHB0KHRjX3ZhbGxfaW5wdXQsIHN1YnNldCA9ICJ2aXNpdGJpcGFydD09J3YxJyIpCnRfdjFfaW5wdXQgPC0gc3Vic2V0X2V4cHQodF92YWxsX2lucHV0LCBzdWJzZXQgPSAidmlzaXRiaXBhcnQ9PSd2MSciKQoKdGNfdmFsbF9pbnB1dCA8LSBub3JtYWxpemVfZXhwdCh0Y192YWxsX2lucHV0LCB0cmFuc2Zvcm0gPSAibG9nMiIsIGNvbnZlcnQgPSAiY3BtIikKdF92YWxsX2lucHV0IDwtIG5vcm1hbGl6ZV9leHB0KHRfdmFsbF9pbnB1dCwgdHJhbnNmb3JtID0gImxvZzIiLCBjb252ZXJ0ID0gImNwbSIpCnRjX3YxX2lucHV0IDwtIG5vcm1hbGl6ZV9leHB0KHRjX3YxX2lucHV0LCB0cmFuc2Zvcm0gPSAibG9nMiIsIGNvbnZlcnQgPSAiY3BtIikKdF92MV9pbnB1dCA8LSBub3JtYWxpemVfZXhwdCh0X3YxX2lucHV0LCB0cmFuc2Zvcm0gPSAibG9nMiIsIGNvbnZlcnQgPSAiY3BtIikKCnRjX3ZhbGxfb3V0Y29tZV9mYWN0b3IgPC0gYXMuZmFjdG9yKGFzLmNoYXJhY3RlcihwRGF0YSh0Y192YWxsX2lucHV0KVtbcmVmX2NvbF1dKSkKdF92YWxsX291dGNvbWVfZmFjdG9yIDwtIGFzLmZhY3Rvcihhcy5jaGFyYWN0ZXIocERhdGEodF92YWxsX2lucHV0KVtbcmVmX2NvbF1dKSkKdGNfdjFfb3V0Y29tZV9mYWN0b3IgPC0gYXMuZmFjdG9yKGFzLmNoYXJhY3RlcihwRGF0YSh0Y192MV9pbnB1dClbW3JlZl9jb2xdXSkpCnRfdjFfb3V0Y29tZV9mYWN0b3IgPC0gYXMuZmFjdG9yKGFzLmNoYXJhY3RlcihwRGF0YSh0X3YxX2lucHV0KVtbcmVmX2NvbF1dKSkKYGBgCgojIyBQZXJmb3JtIG5vcm1hbGl6YXRpb24vZmlsdGVyaW5nIG9wZXJhdGlvbnMgYXMgcGVyIHRoZSBleHBsb3JhdG9yeSBkb2N1bWVudAoKSW4gdGhlIGZvbGxvd2luZyBibG9jayBJIHdpbGwgdXNlIHRoZSBzYW1lIG5vcm1hbGl6YXRpb25zIGFuZCBmaWx0ZXJzCmFzIGFib3ZlIG9uIGVhY2ggb2YgdGhlIGZvdXIgaW5wdXRzLgoKYGBge3J9CnRjX3ZhbGxfbm9ybSA8LSBub3JtYWxpemVfZXhwdCh0Y192YWxsX2lucHV0LCBmaWx0ZXIgPSAiY3YiLCBjdl9taW4gPSAwLjEpCnRjX3YxX25vcm0gPC0gbm9ybWFsaXplX2V4cHQodGNfdjFfaW5wdXQsIGZpbHRlciA9ICJjdiIsIGN2X21pbiA9IDAuMSkKdF92YWxsX25vcm0gPC0gbm9ybWFsaXplX2V4cHQodF92YWxsX2lucHV0LCBmaWx0ZXIgPSAiY3YiLCBjdl9taW4gPSAwLjEpCnRfdjFfbm9ybSA8LSBub3JtYWxpemVfZXhwdCh0X3YxX2lucHV0LCBmaWx0ZXIgPSAiY3YiLCBjdl9taW4gPSAwLjEpCgp0Y192YWxsX3RleHBycyA8LSB0KGV4cHJzKHRjX3ZhbGxfbm9ybSkpCnRjX3YxX3RleHBycyA8LSB0KGV4cHJzKHRjX3YxX25vcm0pKQp0X3ZhbGxfdGV4cHJzIDwtIHQoZXhwcnModF92YWxsX25vcm0pKQp0X3YxX3RleHBycyA8LSB0KGV4cHJzKHRfdjFfbm9ybSkpCgpzZF9nZW5lcyA8LSAzMDAwCnRjX3ZhbGxfc3RkZXYgPC0gYXBwbHkodGNfdmFsbF90ZXhwcnMsIDIsIHNkKQp0Y192YWxsX3ByZWQgPC0gb3JkZXIodGNfdmFsbF9zdGRldiwgZGVjcmVhc2luZyA9IFRSVUUpWzE6c2RfZ2VuZXNdCnRjX3ZhbGxfdGV4cHJzIDwtIHRjX3ZhbGxfdGV4cHJzWywgdGNfdmFsbF9wcmVkXQp0Y192MV9zdGRldiA8LSBhcHBseSh0Y192MV90ZXhwcnMsIDIsIHNkKQp0Y192MV9wcmVkIDwtIG9yZGVyKHRjX3YxX3N0ZGV2LCBkZWNyZWFzaW5nID0gVFJVRSlbMTpzZF9nZW5lc10KdGNfdjFfdGV4cHJzIDwtIHRjX3YxX3RleHByc1ssIHRjX3YxX3ByZWRdCnRfdmFsbF9zdGRldiA8LSBhcHBseSh0X3ZhbGxfdGV4cHJzLCAyLCBzZCkKdF92YWxsX3ByZWQgPC0gb3JkZXIodF92YWxsX3N0ZGV2LCBkZWNyZWFzaW5nID0gVFJVRSlbMTpzZF9nZW5lc10KdF92YWxsX3RleHBycyA8LSB0X3ZhbGxfdGV4cHJzWywgdF92YWxsX3ByZWRdCnRfdjFfc3RkZXYgPC0gYXBwbHkodF92MV90ZXhwcnMsIDIsIHNkKQp0X3YxX3ByZWQgPC0gb3JkZXIodF92MV9zdGRldiwgZGVjcmVhc2luZyA9IFRSVUUpWzE6c2RfZ2VuZXNdCnRfdjFfdGV4cHJzIDwtIHRfdjFfdGV4cHJzWywgdF92MV9wcmVkXQoKdGNfdmFsbF9wcmVwcm9jIDwtIHByZVByb2Nlc3ModGNfdmFsbF90ZXhwcnMsIG1ldGhvZCA9ICJjZW50ZXIiKQp0Y192YWxsX3RleHBycyA8LSBwcmVkaWN0KHRjX3ZhbGxfcHJlcHJvYywgdGNfdmFsbF90ZXhwcnMpCnRjX3YxX3ByZXByb2MgPC0gcHJlUHJvY2Vzcyh0Y192MV90ZXhwcnMsIG1ldGhvZCA9ICJjZW50ZXIiKQp0Y192MV90ZXhwcnMgPC0gcHJlZGljdCh0Y192MV9wcmVwcm9jLCB0Y192MV90ZXhwcnMpCnRfdmFsbF9wcmVwcm9jIDwtIHByZVByb2Nlc3ModF92YWxsX3RleHBycywgbWV0aG9kID0gImNlbnRlciIpCnRfdmFsbF90ZXhwcnMgPC0gcHJlZGljdCh0X3ZhbGxfcHJlcHJvYywgdF92YWxsX3RleHBycykKdF92MV9wcmVwcm9jIDwtIHByZVByb2Nlc3ModF92MV90ZXhwcnMsIG1ldGhvZCA9ICJjZW50ZXIiKQp0X3YxX3RleHBycyA8LSBwcmVkaWN0KHRfdjFfcHJlcHJvYywgdF92MV90ZXhwcnMpCgp0Y192YWxsX3ByZXByb2MgPC0gcHJlUHJvY2Vzcyh0Y192YWxsX3RleHBycywgbWV0aG9kID0gImNvcnIiLCBjdXRvZmYgPSAwLjk1KQp0Y192YWxsX3RleHBycyA8LSBwcmVkaWN0KHRjX3ZhbGxfcHJlcHJvYywgdGNfdmFsbF90ZXhwcnMpCnRjX3YxX3ByZXByb2MgPC0gcHJlUHJvY2Vzcyh0Y192MV90ZXhwcnMsIG1ldGhvZCA9ICJjb3JyIiwgY3V0b2ZmID0gMC45NSkKdGNfdjFfdGV4cHJzIDwtIHByZWRpY3QodGNfdjFfcHJlcHJvYywgdGNfdjFfdGV4cHJzKQp0X3ZhbGxfcHJlcHJvYyA8LSBwcmVQcm9jZXNzKHRfdmFsbF90ZXhwcnMsIG1ldGhvZCA9ICJjb3JyIiwgY3V0b2ZmID0gMC45NSkKdF92YWxsX3RleHBycyA8LSBwcmVkaWN0KHRfdmFsbF9wcmVwcm9jLCB0X3ZhbGxfdGV4cHJzKQp0X3YxX3ByZXByb2MgPC0gcHJlUHJvY2Vzcyh0X3YxX3RleHBycywgbWV0aG9kID0gImNvcnIiLCBjdXRvZmYgPSAwLjk1KQp0X3YxX3RleHBycyA8LSBwcmVkaWN0KHRfdjFfcHJlcHJvYywgdF92MV90ZXhwcnMpCgpjaG9zZW5fZmFjdG9ycyA8LSBjKCJmaW5hbG91dGNvbWUiLCAiZG9ub3IiLCAicGVyc2lzdGVuY2UiLCAidmlzaXRudW1iZXIiLAogICAgICAgICAgICAgICAgICAgICJzZWxlY3Rpb25tZXRob2QiLCAidHlwZW9mY2VsbHMiLCAidGltZSIpCnRjX3ZhbGxfbWV0YSA8LSBwRGF0YSh0Y192YWxsX25vcm0pWywgY2hvc2VuX2ZhY3RvcnNdCnRjX3YxX21ldGEgPC0gcERhdGEodGNfdjFfbm9ybSlbLCBjaG9zZW5fZmFjdG9yc10KdF92YWxsX21ldGEgPC0gcERhdGEodF92YWxsX25vcm0pWywgY2hvc2VuX2ZhY3RvcnNdCnRfdjFfbWV0YSA8LSBwRGF0YSh0X3YxX25vcm0pWywgY2hvc2VuX2ZhY3RvcnNdCgpyZWZfY29sIDwtICJmaW5hbG91dGNvbWUiCnRjX3ZhbGxfb3V0Y29tZSA8LSBhcy5mYWN0b3IoYXMuY2hhcmFjdGVyKHBEYXRhKHRjX3ZhbGxfbm9ybSlbW3JlZl9jb2xdXSkpCnRjX3YxX291dGNvbWUgPC0gYXMuZmFjdG9yKGFzLmNoYXJhY3RlcihwRGF0YSh0Y192MV9ub3JtKVtbcmVmX2NvbF1dKSkKdF92YWxsX291dGNvbWUgPC0gYXMuZmFjdG9yKGFzLmNoYXJhY3RlcihwRGF0YSh0X3ZhbGxfbm9ybSlbW3JlZl9jb2xdXSkpCnRfdjFfb3V0Y29tZSA8LSBhcy5mYWN0b3IoYXMuY2hhcmFjdGVyKHBEYXRhKHRfdjFfbm9ybSlbW3JlZl9jb2xdXSkpCgp0Y192YWxsX21sX2RmIDwtIGNiaW5kLmRhdGEuZnJhbWUodGNfdmFsbF9vdXRjb21lLCBhcy5kYXRhLmZyYW1lKHRjX3ZhbGxfdGV4cHJzKSkKdGNfdmFsbF9tbF9kZltbcmVmX2NvbF1dIDwtIGFzLmZhY3Rvcih0Y192YWxsX21sX2RmW1sidGNfdmFsbF9vdXRjb21lIl1dKQp0Y192MV9tbF9kZiA8LSBjYmluZC5kYXRhLmZyYW1lKHRjX3YxX291dGNvbWUsIGFzLmRhdGEuZnJhbWUodGNfdjFfdGV4cHJzKSkKdGNfdjFfbWxfZGZbW3JlZl9jb2xdXSA8LSBhcy5mYWN0b3IodGNfdjFfbWxfZGZbWyJ0Y192MV9vdXRjb21lIl1dKQp0X3ZhbGxfbWxfZGYgPC0gY2JpbmQuZGF0YS5mcmFtZSh0X3ZhbGxfb3V0Y29tZSwgYXMuZGF0YS5mcmFtZSh0X3ZhbGxfdGV4cHJzKSkKdF92YWxsX21sX2RmW1tyZWZfY29sXV0gPC0gYXMuZmFjdG9yKHRfdmFsbF9tbF9kZltbInRfdmFsbF9vdXRjb21lIl1dKQp0X3YxX21sX2RmIDwtIGNiaW5kLmRhdGEuZnJhbWUodF92MV9vdXRjb21lLCBhcy5kYXRhLmZyYW1lKHRfdjFfdGV4cHJzKSkKdF92MV9tbF9kZltbcmVmX2NvbF1dIDwtIGFzLmZhY3Rvcih0X3YxX21sX2RmW1sidF92MV9vdXRjb21lIl1dKQpgYGAKCiMjIEl0ZXJhdGUgVHVtYWNvK0NhbGkgYWxsIHZpc2l0cwoKYGBge3IsIGV2YWw9RkFMU0V9CnRjX3ZhbGxfc3VtbWFyeV94bHN4IDwtIGdsdWUoImV4Y2VsL3RjX3ZhbGxfbWxfc3VtbWFyeS12e3Zlcn0ueGxzeCIpCnRjX3ZhbGxfa25uIDwtIGNsYXNzaWZ5X25fdGltZXModGNfdmFsbF90ZXhwcnMsIHRjX3ZhbGxfbWV0YSwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBvdXRjb21lX2NvbHVtbiA9IHJlZl9jb2wsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgbWV0aG9kID0gImtubiIsIHNhbXBsZXIgPSAiY3YiKQp3cml0dGVuIDwtIHdyaXRlX2NsYXNzaWZpZXJfc3VtbWFyeSh0Y192YWxsX2tubiwgZXhjZWwgPSB0Y192YWxsX3N1bW1hcnlfeGxzeCkKdGNfdmFsbF9nYiA8LSBjbGFzc2lmeV9uX3RpbWVzKHRjX3ZhbGxfdGV4cHJzLCB0Y192YWxsX21ldGEsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBvdXRjb21lX2NvbHVtbiA9IHJlZl9jb2wsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBtZXRob2QgPSAieGdiVHJlZSIsIHNhbXBsZXIgPSAiY3YiKQp3cml0dGVuIDwtIHdyaXRlX2NsYXNzaWZpZXJfc3VtbWFyeSh0Y192YWxsX2diLCBleGNlbCA9IHdyaXR0ZW5bWyJ3YiJdXSkKdGNfdmFsbF9nbG0gPC0gY2xhc3NpZnlfbl90aW1lcyh0Y192YWxsX3RleHBycywgdGNfdmFsbF9tZXRhLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIG91dGNvbWVfY29sdW1uID0gcmVmX2NvbCwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBtZXRob2QgPSAiZ2xtbmV0Iiwgc2FtcGxlciA9ICJjdiIpCndyaXR0ZW4gPC0gd3JpdGVfY2xhc3NpZmllcl9zdW1tYXJ5KHRjX3ZhbGxfZ2xtLCBleGNlbCA9IHdyaXR0ZW5bWyJ3YiJdXSkKdGNfdmFsbF9yZiA8LSBjbGFzc2lmeV9uX3RpbWVzKHRjX3ZhbGxfdGV4cHJzLCB0Y192YWxsX21ldGEsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBvdXRjb21lX2NvbHVtbiA9IHJlZl9jb2wsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBtZXRob2QgPSAicmFuZ2VyIiwgc2FtcGxlciA9ICJjdiIpCndyaXR0ZW4gPC0gd3JpdGVfY2xhc3NpZmllcl9zdW1tYXJ5KHRjX3ZhbGxfcmYsIGV4Y2VsID0gd3JpdHRlbltbIndiIl1dKQpvcGVueGxzeDo6c2F2ZVdvcmtib29rKHdyaXR0ZW5bWyJ3YiJdXSwgZmlsZSA9IHRjX3ZhbGxfc3VtbWFyeV94bHN4KQpgYGAKCiMjIEl0ZXJhdGUgVHVtYWNvK0NhbGkgVmlzaXQgMQoKYGBge3IsIGV2YWw9RkFMU0V9CnRjX3YxX3N1bW1hcnlfeGxzeCA8LSBnbHVlKCJleGNlbC90Y192MV9tbF9zdW1tYXJ5LXZ7dmVyfS54bHN4IikKdGNfdjFfa25uIDwtIGNsYXNzaWZ5X25fdGltZXModGNfdjFfdGV4cHJzLCB0Y192MV9tZXRhLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIG91dGNvbWVfY29sdW1uID0gcmVmX2NvbCwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBtZXRob2QgPSAia25uIiwgc2FtcGxlciA9ICJjdiIpCndyaXR0ZW4gPC0gd3JpdGVfY2xhc3NpZmllcl9zdW1tYXJ5KHRjX3YxX2tubiwgZXhjZWwgPSB0Y192MV9zdW1tYXJ5X3hsc3gpCnRjX3YxX2diIDwtIGNsYXNzaWZ5X25fdGltZXModGNfdjFfdGV4cHJzLCB0Y192MV9tZXRhLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgb3V0Y29tZV9jb2x1bW4gPSByZWZfY29sLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgbWV0aG9kID0gInhnYlRyZWUiLCBzYW1wbGVyID0gImN2IikKd3JpdHRlbiA8LSB3cml0ZV9jbGFzc2lmaWVyX3N1bW1hcnkodGNfdjFfZ2IsIGV4Y2VsID0gd3JpdHRlbltbIndiIl1dKQp0Y192MV9nbG0gPC0gY2xhc3NpZnlfbl90aW1lcyh0Y192MV90ZXhwcnMsIHRjX3YxX21ldGEsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgb3V0Y29tZV9jb2x1bW4gPSByZWZfY29sLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIG1ldGhvZCA9ICJnbG1uZXQiLCBzYW1wbGVyID0gImN2IikKd3JpdHRlbiA8LSB3cml0ZV9jbGFzc2lmaWVyX3N1bW1hcnkodGNfdjFfZ2xtLCBleGNlbCA9IHdyaXR0ZW5bWyJ3YiJdXSkKdGNfdjFfcmYgPC0gY2xhc3NpZnlfbl90aW1lcyh0Y192MV90ZXhwcnMsIHRjX3YxX21ldGEsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBvdXRjb21lX2NvbHVtbiA9IHJlZl9jb2wsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBtZXRob2QgPSAicmFuZ2VyIiwgc2FtcGxlciA9ICJjdiIpCndyaXR0ZW4gPC0gd3JpdGVfY2xhc3NpZmllcl9zdW1tYXJ5KHRjX3YxX3JmLCBleGNlbCA9IHdyaXR0ZW5bWyJ3YiJdXSkKb3Blbnhsc3g6OnNhdmVXb3JrYm9vayh3cml0dGVuW1sid2IiXV0sIGZpbGUgPSB0Y192MV9zdW1tYXJ5X3hsc3gpCmBgYAoKIyMgVHVtYWNvIGFsbCB2aXNpdHMKCmBgYHtyLCBldmFsPUZBTFNFfQp0X3ZhbGxfc3VtbWFyeV94bHN4IDwtIGdsdWUoImV4Y2VsL3RfdmFsbF9tbF9zdW1tYXJ5LXZ7dmVyfS54bHN4IikKdF92YWxsX2tubiA8LSBjbGFzc2lmeV9uX3RpbWVzKHRfdmFsbF90ZXhwcnMsIHRfdmFsbF9tZXRhLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIG91dGNvbWVfY29sdW1uID0gcmVmX2NvbCwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBtZXRob2QgPSAia25uIiwgc2FtcGxlciA9ICJjdiIpCndyaXR0ZW4gPC0gd3JpdGVfY2xhc3NpZmllcl9zdW1tYXJ5KHRfdmFsbF9rbm4sIGV4Y2VsID0gdF92YWxsX3N1bW1hcnlfeGxzeCkKdF92YWxsX2diIDwtIGNsYXNzaWZ5X25fdGltZXModF92YWxsX3RleHBycywgdF92YWxsX21ldGEsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBvdXRjb21lX2NvbHVtbiA9IHJlZl9jb2wsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBtZXRob2QgPSAieGdiVHJlZSIsIHNhbXBsZXIgPSAiY3YiKQp3cml0dGVuIDwtIHdyaXRlX2NsYXNzaWZpZXJfc3VtbWFyeSh0X3ZhbGxfZ2IsIGV4Y2VsID0gd3JpdHRlbltbIndiIl1dKQp0X3ZhbGxfZ2xtIDwtIGNsYXNzaWZ5X25fdGltZXModF92YWxsX3RleHBycywgdF92YWxsX21ldGEsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgb3V0Y29tZV9jb2x1bW4gPSByZWZfY29sLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIG1ldGhvZCA9ICJnbG1uZXQiLCBzYW1wbGVyID0gImN2IikKd3JpdHRlbiA8LSB3cml0ZV9jbGFzc2lmaWVyX3N1bW1hcnkodF92YWxsX2dsbSwgZXhjZWwgPSB3cml0dGVuW1sid2IiXV0pCnRfdmFsbF9yZiA8LSBjbGFzc2lmeV9uX3RpbWVzKHRfdmFsbF90ZXhwcnMsIHRfdmFsbF9tZXRhLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgb3V0Y29tZV9jb2x1bW4gPSByZWZfY29sLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgbWV0aG9kID0gInJhbmdlciIsIHNhbXBsZXIgPSAiY3YiKQp3cml0dGVuIDwtIHdyaXRlX2NsYXNzaWZpZXJfc3VtbWFyeSh0X3ZhbGxfcmYsIGV4Y2VsID0gd3JpdHRlbltbIndiIl1dKQpvcGVueGxzeDo6c2F2ZVdvcmtib29rKHdyaXR0ZW5bWyJ3YiJdXSwgZmlsZSA9IHRfdmFsbF9zdW1tYXJ5X3hsc3gpCmBgYAoKIyMgVHVtYWNvIFZpc2l0IDEKCmBgYHtyLCBldmFsPUZBTFNFfQp0X3YxX3N1bW1hcnlfeGxzeCA8LSBnbHVlKCJleGNlbC90X3YxX21sX3N1bW1hcnktdnt2ZXJ9Lnhsc3giKQp0X3YxX2tubiA8LSBjbGFzc2lmeV9uX3RpbWVzKHRfdjFfdGV4cHJzLCB0X3YxX21ldGEsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgb3V0Y29tZV9jb2x1bW4gPSByZWZfY29sLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIG1ldGhvZCA9ICJrbm4iLCBzYW1wbGVyID0gImN2IikKd3JpdHRlbiA8LSB3cml0ZV9jbGFzc2lmaWVyX3N1bW1hcnkodF92MV9rbm4sIGV4Y2VsID0gdF92MV9zdW1tYXJ5X3hsc3gpCnRfdjFfZ2IgPC0gY2xhc3NpZnlfbl90aW1lcyh0X3YxX3RleHBycywgdF92MV9tZXRhLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgb3V0Y29tZV9jb2x1bW4gPSByZWZfY29sLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgbWV0aG9kID0gInhnYlRyZWUiLCBzYW1wbGVyID0gImN2IikKd3JpdHRlbiA8LSB3cml0ZV9jbGFzc2lmaWVyX3N1bW1hcnkodF92MV9nYiwgZXhjZWwgPSB3cml0dGVuW1sid2IiXV0pCnRfdjFfZ2xtIDwtIGNsYXNzaWZ5X25fdGltZXModF92MV90ZXhwcnMsIHRfdjFfbWV0YSwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBvdXRjb21lX2NvbHVtbiA9IHJlZl9jb2wsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgbWV0aG9kID0gImdsbW5ldCIsIHNhbXBsZXIgPSAiY3YiKQp3cml0dGVuIDwtIHdyaXRlX2NsYXNzaWZpZXJfc3VtbWFyeSh0X3YxX2dsbSwgZXhjZWwgPSB3cml0dGVuW1sid2IiXV0pCnRfdjFfcmYgPC0gY2xhc3NpZnlfbl90aW1lcyh0X3YxX3RleHBycywgdF92MV9tZXRhLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgb3V0Y29tZV9jb2x1bW4gPSByZWZfY29sLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgbWV0aG9kID0gInJhbmdlciIsIHNhbXBsZXIgPSAiY3YiKQp3cml0dGVuIDwtIHdyaXRlX2NsYXNzaWZpZXJfc3VtbWFyeSh0X3YxX3JmLCBleGNlbCA9IHdyaXR0ZW5bWyJ3YiJdXSkKb3Blbnhsc3g6OnNhdmVXb3JrYm9vayh3cml0dGVuW1sid2IiXV0sIGZpbGUgPSB0X3YxX3N1bW1hcnlfeGxzeCkKYGBgCgojIEJpYmxpb2dyYXBoeQo=