- 6640 Views
- 1 replies
- 1 kudos
Resolved! ForeachBatch() - Get results from batchDF._jdf.sparkSession().sql('merge stmt')
Most python examples show the structure of the foreachBatch method as:def foreachBatchFunc(batchDF, batchId): batchDF.createOrReplaceTempView('viewName') ( batchDF ._jdf.sparkSession() .sql( ...
- 6640 Views
- 1 replies
- 1 kudos
- 1 kudos
Just found a solution...Need to convert the Java Dataframe (jdf) to a DataFramefrom pyspark import sql def batchFunc(batchDF, batchId): batchDF.createOrReplaceTempView('viewName') sparkSession = batchDF._jdf.sparkSession() resJdf = sparkSes...
- 1 kudos
- 4630 Views
- 2 replies
- 5 kudos
Resolved! Autoloader creates columns not present in the source
I have been exploring Autoloader to ingest gzipped JSON files from an S3 source.The notebook fails in the first run due to schema mismatch, after re-running the notebook, the schema evolves and the ingestion runs successfully.On analysing the schema ...
- 4630 Views
- 2 replies
- 5 kudos
- 5 kudos
Hi @Debayan Mukherjee , @Kaniz Fatma Thank you for replying to my question.I was able to figure out the issue. I was creating the schema and checkpoint folders in the same path as the source location for the autoloader. This caused the schema to ch...
- 5 kudos
- 5619 Views
- 2 replies
- 0 kudos
SUBNET_EXHAUSTED_FAILURE(CLOUD_FAILURE): or No more address space to create NIC within injected virtual network
Currently we are using an all-purpose compute cluster. When we tried to allocate the scheduled jobs to job cluster, we are blocked at the following error:SUBNET_EXHAUSTED_FAILURE(CLOUD_FAILURE): azure_error_code:SubnetIsFull,azure_error_message:No mo...
- 5619 Views
- 2 replies
- 0 kudos
- 0 kudos
Answering your questions - yes, your vnet/subnet is out of non-occupied IPs and this can be fixed by allocating more IPs to your network address space.Each cluster requires it's own IP, so if there are none available, it simply cannot start.
- 0 kudos
- 3530 Views
- 2 replies
- 3 kudos
Is it possible to create a feature store training set directly from a feature store table?
Rather than joining features from different tables, I just wanted to use a single feature store table and select some of its features, but still log the model in the feature store. The problem I am facing is that I do not know how to create the train...
- 3530 Views
- 2 replies
- 3 kudos
- 3 kudos
Hi, Could you please refer https://docs.databricks.com/machine-learning/feature-store/train-models-with-feature-store.html#create-a-trainingset-using-the-same-feature-multiple-times and let us know if this helps.
- 3 kudos
- 3689 Views
- 2 replies
- 1 kudos
MEMORY_ONLY not working
val doubledAmount = premiumCustomers.map(x=>(x._1, x._2*2)).persist(StorageLevel.MEMORY_ONLY) error: not found: value StorageLevel
- 3689 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @Gaurav Poojary ,Can you please try the below as displayed in the image it is working for me without any issues.Happy Learning!!
- 1 kudos
- 3601 Views
- 3 replies
- 3 kudos
Set taskValues in DLT workbooks
Is "setting taskValues in DLT workbooks" supported?I tried setting a task value in a DLT workbook, but it does not seem supported, so downstream workbooks within the same workflows job cannot consume this task value.
- 3601 Views
- 3 replies
- 3 kudos
- 3 kudos
I have the same issue, I also want to know databricks support taskValue between taskJob and DLT or not?
- 3 kudos
- 13491 Views
- 3 replies
- 5 kudos
Some very simple functions in Pandas on Spark are very slow
I have a pandas on spark dataframe with 8 million rows and 20 columns. It took 3.48 minutes to run df.shape and it takes. It also takes a long time to run df.head took 4.55 minutes . By contrast df.var1.value_counts().reset_index() took only 0.18 sec...
- 13491 Views
- 3 replies
- 5 kudos
- 5 kudos
The reason why this is slow is because pandas needs an index column to perform `shape` or `head`. If you don't provide one, pyspark pandas enumerates the entire dataframe to create a default one. For example, given columns A, B, and C in dataframe `d...
- 5 kudos
- 8661 Views
- 2 replies
- 1 kudos
Vnet peering settings is not enable in Azure databricks premium , even though its deployed inside my VNET?
Hi All,Vnet peering settings is not enabled in Azure databricks , even though its deployed inside my VNET?Here i not mentioned my vnet and subnet details , but filled this and created databricks (without private endpoint - allow public access)virtual...
- 8661 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi, VNET peering is not supported or possible on VNET-injected workspaces. Please refer: https://learn.microsoft.com/en-us/azure/databricks/administration-guide/cloud-configurations/azure/vnet-peering#requirements
- 1 kudos
- 4785 Views
- 2 replies
- 2 kudos
load new data in delta table
Hello all,I want to know how to update new data in delta table from new csv file.here is the code that i have used to create delta table from csv file and loaded data. but i have go new updated file and trying to load new data but not able to any gui...
- 4785 Views
- 2 replies
- 2 kudos
- 2 kudos
Thank you, i tried that and it ended in error, the table created with delta are from csv which must have converted to parquet file and all the columns are varchar or string. so not if i want to entered new file it ends in incmopatibility error for da...
- 2 kudos
- 22391 Views
- 8 replies
- 10 kudos
Resolved! How i can add ADLS Gen2 - OAuth 2.0 as Cluster scope for my High concurrency Shared Cluster (without unity catalog)?
Hi All,Kindly help me , how i can add the ADLS gen2 OAuth 2.0 authentication to my high concurrency shared cluster. I want to scope this authentication to entire cluster not for particular notebook.Currently i have added them as spark configuration o...
- 22391 Views
- 8 replies
- 10 kudos
- 10 kudos
error is because of missing default settings (create new cluster and do not remove them),the warning is because secrets should be put in secret scope, and then you should reference secrets in settings
- 10 kudos
- 10262 Views
- 2 replies
- 1 kudos
Resolved! How to extract source data from on-premise databases into a data lake and load with AutoLoader?
Here is the situation I am working with. I am trying to extract source data using Databricks JDBC connector using SQL Server databases as my data source. I want to write those into a directory in my data lake as JSON files, then have AutoLoader ing...
- 10262 Views
- 2 replies
- 1 kudos
- 1 kudos
To add to @werners point, I would use ADF to load SQL server data into ADLS Gen 2 as json. Then Load these Raw Json files from your ADLS base location into a Delta table using Autoloader.Delta Live Tables can be used in this scenario.You can also reg...
- 1 kudos
- 1981 Views
- 1 replies
- 0 kudos
Resolved! How to create border for sme specific cells?
i tried some code to create border for excel sheet, for particular cell iam able to write but while i am trying with some set of cells means it is showing error.
- 1981 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Mohammed sadamusean ,Can you please try similar to below code using loops, I have implemented a similar use case that might be useful, please let me know if you need further top = Side(border_style = 'thin',color = '00000000') bottom = Side(bor...
- 0 kudos
- 7034 Views
- 4 replies
- 7 kudos
Resolved! Getting status of "If Condition" Activity into a variable
"If Condition" has lot of activities that can succeeded or fail. If any activity fails then whole "If Condition" fails. I have to get the status of the "If Condition" activity (pass or fail) so that i can use it for processing in the next notebook t...
- 7034 Views
- 4 replies
- 7 kudos
- 7 kudos
In your ADF Pipeline activity, set two different pipeline activities from the If condition activity based on success or failure (the green and red arrows). Then inside each pipeline activity, you can add a set variable, get variable and your adb note...
- 7 kudos
- 3482 Views
- 2 replies
- 2 kudos
Databricks JAR task type functionality
Hi, I would like to understand Databricks JAR based workflow tasks. Can I interpret JAR based runs to be something like a spark-submit on a cluster? In the logs, I was expecting to see the spark-submit --class com.xyz --num-executors 4 etc., And, the...
- 3482 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi, I did try using the Workflows>Jobs>CreateTask>JarTaskType>UploadedMyJAR and Class and created JobCluster and tested this task. This JAR reads some tables as input, does some transformations and output as writing some other tables. I would like t...
- 2 kudos
- 9238 Views
- 1 replies
- 1 kudos
Partition filter is skipped when table is used in where condition, why?
Hi,maybe someone can help me i do want to run very narrow query SELECT * FROM my_table WHERE snapshot_date IN('2023-01-06', '2023-01-07') -- part of the physical plan: -- Location: PreparedDeltaFileIndex [dbfs:/...] -- PartitionFilters: [cast(snaps...
- 9238 Views
- 1 replies
- 1 kudos
- 1 kudos
No hints on partition pruning afaik.The reason the partitions were not pruned is because the second query generates a completely different plan.To be able to filter the partitions, a join first has to happen. And in this case it means the table has...
- 1 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 586 | |
| 349 | |
| 287 |
