- 4482 Views
- 2 replies
- 1 kudos
SQL Analytics Map Visualization: Map marker size
Hello all, I am trying to use the Map visualization in SQL Analytics Dashboard in Databricks. Does any one knows how or if we can change the size/radius of the markers based on values in another column. This seems like a very trivial parameter but I ...
- 4482 Views
- 2 replies
- 1 kudos
- 7262 Views
- 5 replies
- 1 kudos
Autoloader and "cleanSource"
Hi All, We are trying to use the Spark 3 structured streaming feature/option ".option('cleanSource','archive')" to archive processed files. This is working as expected using the standard spark implementation, however does not appear to work using aut...
- 7262 Views
- 5 replies
- 1 kudos
- 1 kudos
https://docs.databricks.com/ingestion/auto-loader/options.html#common-auto-loader-optionscleanSource is not a listed option so it won't do anything.Maybe event retention is something you can use?
- 1 kudos
- 8890 Views
- 3 replies
- 3 kudos
Is there a way to CONCAT two dataframes on either of the axis (row/column) and transpose the dataframe in PySpark?
I'm reshaping my dataframe as per requirement and I came across this situation where I'm concatenating 2 dataframes and then transposing them. I've done this previously using pandas and the syntax for pandas goes as below:import pandas as pd df1 = ...
- 8890 Views
- 3 replies
- 3 kudos
- 3 kudos
Hi @Kaniz Fatma​ ,I no longer see the answer you've posted, but I see you were suggesting to use `union`. As per my understanding, union are used to stack the dfs one upon another with similar schema / column names.In my situation, I have 2 different...
- 3 kudos
- 2141 Views
- 1 replies
- 0 kudos
Workflow Pipeline in Azure Databrick is throwing error for EventHubsSourceProvider could not be instantiated
I am using the sample code which is available in getting start tutorial. And it is simple read the json file and move in another table. But it is throwing error related to EventHubsSourceProvider
- 2141 Views
- 1 replies
- 0 kudos
- 10864 Views
- 3 replies
- 6 kudos
Is there any way to overwrite a partition in delta table without specifying each and every partition in replace where? For non dated partitions, this is really a mess with delta tables.
Is there any way to overwrite a partition in delta table without specifying each and every partition in replace where. For non dated partitions, this is really a mess with delta tables.Most of my DE teams don't want to adopt delta because of these gl...
- 10864 Views
- 3 replies
- 6 kudos
- 6 kudos
Hi @Saurabh Verma​ following up did you get a chance to check @Hubert Dudek​ previous comments ?
- 6 kudos
- 2843 Views
- 1 replies
- 1 kudos
Query silently failed
Hello all, I'm using the older 6.4 runtime and noticed that a query return no result whereas the same query on 10.4 provided the expected result. This is bad, because I got no error, simply no result at all.Is there is some spark settings on the clus...
- 2843 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @Alessio Palma​ following up did you get chance to check @Kaniz Fatma​ 's previous comments ?
- 1 kudos
- 6910 Views
- 1 replies
- 1 kudos
Append an empty dataframe to a list of dataframes using for loop in python
I have the following 3 dataframes:I want to append df_forecast to each of df2_CA and df2_USA using a for-loop. However when I run my code, df_forecast is not appending: df2_CA and df2_USA appear exactly as shown above.Here’s the code:df_list=[df2_CA,...
- 6910 Views
- 1 replies
- 1 kudos
- 1 kudos
@Jack Homareau​ Can you try union functionality with dataframes?https://sparkbyexamples.com/pyspark/pyspark-union-and-unionall/and then try to fill NaNs with the desired values?
- 1 kudos
- 7359 Views
- 4 replies
- 2 kudos
Error using Synapse ML: JavaPackage object is not callable
I am using DBR version 10.1. I want to use Synapse ML package. I am able to install and import it by following instructions on the link: https://github.com/microsoft/SynapseML. However when I try to run the code it gives me the error shown in the att...
- 7359 Views
- 4 replies
- 2 kudos
- 2 kudos
Hello @Vikram Mahawal​ Clusters need to be in the running state to install/uninstall the libraries. Could you please start the cluster and try installing it.If you are still stuck, please file a support case with us, so we can take a look.Thanks
- 2 kudos
- 4693 Views
- 3 replies
- 1 kudos
Resolved! Connect from Databricks to Hbase HDinsight cluster.
Hi, I have Databricks installation in Azure. I want to run a job that connects to HBase in a separate HDinsight cluster.What I tried:Created a peering between base cluster and Databricks vNets.I can ping IPs of Hbase zookeeper nodes but I cannot acce...
- 4693 Views
- 3 replies
- 1 kudos
- 2559 Views
- 2 replies
- 2 kudos
Merge into and data loss
I have a delta table with 20 M rows, Ther table is being updated dozens of times per day. The merge into is used, and the merge works fine for 1 year. But recently I begin notice some of data is deleted from merge into without delete specified. Mer...
- 2559 Views
- 2 replies
- 2 kudos
- 2 kudos
I can't reproduce the issue anymore. for now, I am going to limit the number of merge into commands as intermediate data transformation does not need versioning history. I am going to try to use combined views for each step, and do a one-time merge i...
- 2 kudos
- 7207 Views
- 1 replies
- 1 kudos
Resolved! Insert query fails with error "The query is not executed because it tries to launch ***** tasks in a single stage, while maximum allowed tasks one query can launch is 100000;
Py4JJavaError: An error occurred while calling o236.sql. : org.apache.spark.SparkException: Job aborted. at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:201) at org.apache.spark.sql.execution.datasources.I...
- 7207 Views
- 1 replies
- 1 kudos
- 1 kudos
could you please increase the below config (at the cluster level) to a higher value or set it to zero spark.databricks.queryWatchdog.maxQueryTasks 0The spark config while it alleviates the issue.
- 1 kudos
- 45453 Views
- 6 replies
- 10 kudos
How do I prevent _success and _committed files in my write output?
Is there a way to prevent the _success and _committed files in my output. It's a tedious task to navigate to all the partitions and delete the files. Note : Final output is stored in Azure ADLS
- 45453 Views
- 6 replies
- 10 kudos
- 10 kudos
Please find the below steps to remove _SUCCESS, _committed and _started files.spark.conf.set("spark.databricks.io.directoryCommit.createSuccessFile","false") to remove success file.run vacuum command multiple times until _committed and _started files...
- 10 kudos
- 4510 Views
- 3 replies
- 1 kudos
dbutils.notebook.run() fails with job aborted but running the notebook individually works
I have a notebook that runs many notebooks in order, along the lines of:```%pythonnotebook_list = ['Notebook1', 'Notebook2'] for notebook in notebook_list: print(f"Now on Notebook: {notebook}") try: dbutils.notebook.run(f'{notebook}', 3600) e...
- 4510 Views
- 3 replies
- 1 kudos
- 1 kudos
I found the problem. Even if a notebook creates and specifies a widget fully, the notebook run process, e.g, dbutils.notebook.run('notebook') will not know how to use it. If I replace my widget with a non-widget provided value, the process works fine...
- 1 kudos
- 978 Views
- 0 replies
- 0 kudos
www.pieseford.ro
Hi, my name is Jerry Maguire and I`m automatic engineer at Piese Ford. Piese originale Ford Fiesta 2008-2012
- 978 Views
- 0 replies
- 0 kudos
- 7328 Views
- 4 replies
- 2 kudos
Resolved! How to view the SQL Query History of traditional Databricks cluster (not Databricks SQL)?
I tried use the Spark Cluster UI. But the queries are truncated.
- 7328 Views
- 4 replies
- 2 kudos
- 2 kudos
In Apache Spark prior to 2.1, once a SQL query was run, there was no way to re-run it; all history was lost. Spark SQL introduced the "replay" functionality in Spark 2.1.0, enabling users to re-run any query they have already run. You can run a query...
- 2 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 580 | |
| 349 | |
| 287 |
