- 7675 Views
- 4 replies
- 4 kudos
Datepicker widget
There are textbox and dropdown list widgets in Databricks. Is there any datepicker widget? If not, is there any plan to add it?
- 7675 Views
- 4 replies
- 4 kudos
- 4 kudos
@Reza Rajabi​ , No, it is not I think at some meeting, someone discussed it. We can ask about it during the following office hours https://databricks.com/p/webinar/databricks-office-hours?utm_source=databricks&utm_medium=email&utm_campaign=7013f0000...
- 4 kudos
- 24298 Views
- 3 replies
- 2 kudos
Resolved! get a "Py4JJavaError: An error occurred while calling o5082.csv." when trying to save to csv file.
Hi, I'm trying to save a dataframe to csv with the code below:output.coalesce(1).write.mode('overwrite').option('header', 'true').csv(tmp_file_path) But it get "Py4JJavaError: An error occurred while calling o5082.csv." error. Any idea how to solve...
- 24298 Views
- 3 replies
- 2 kudos
- 13496 Views
- 4 replies
- 4 kudos
Resolved! Bucketing on Delta Tables
getting error as below while creating buckets on delta table.Error in SQL statement: AnalysisException: Delta bucketed tables are not supported.have fall back to parquet table due to this for some use cases. is their any alternative for this. i have...
- 13496 Views
- 4 replies
- 4 kudos
- 4 kudos
Hi @Rahul Samant​ , we checked internally on this due to certain limitations bucketing is not supported on delta tables, the only alternative for bucketing is to leverage the z ordering, below is the link for reference https://docs.databricks.com/de...
- 4 kudos
- 5529 Views
- 3 replies
- 2 kudos
Resolved! Spark Streaming - only process new files in streaming path?
In our streaming jobs, we currently run streaming (cloudFiles format) on a directory with sales transactions coming every 5 minutes.In this directory, the transactions are ordered in the following format:<streaming-checkpoint-root>/<transaction_date>...
- 5529 Views
- 3 replies
- 2 kudos
- 2 kudos
Update:Seems that maxFileAge was not a good idea. The following with the option "includeExistingFiles" = False solved my problem:streaming_df = ( spark.readStream.format("cloudFiles") .option("cloudFiles.format", extension) .option("...
- 2 kudos
- 5746 Views
- 3 replies
- 4 kudos
Resolved! Azure Databrick SQL bulk insert to AZ SQL
Env: Azure Databrick :version : 9.1 LTS (includes Apache Spark 3.1.2, Scala 2.12)Work Type : 56 GB Memory 2-8 node ( standard D13_V2)No of rows : 2470350 and 115 Column Size : 2.2 GBTime taken approx. 9 min Python Code .What will be best approach for...
- 5746 Views
- 3 replies
- 4 kudos
- 4855 Views
- 4 replies
- 3 kudos
Resolved! uninstalled libraries continue to get installed on cluster startup
We have been trying to update some library versions by uninstalling the old versions and installing new ones. However, the old libraries continue to get installed on cluster startup despite not showing up in the "libraries" tab of the cluster page. W...
- 4855 Views
- 4 replies
- 3 kudos
- 3 kudos
The issue seemed to go away on its own. At some point the libraries page started showing what was getting installed to the cluster, and removing libraries from the page caused them to stop getting installed on cluster startup. I'm guessing there was ...
- 3 kudos
- 4064 Views
- 4 replies
- 7 kudos
Resolved! Increase input rate in Delta Live Tables
Hi,I need to ingest 60 millions json files from S3 and have create a Delta Live Tables to ingest these data to delta table with Auto Loader. However the input rate in my DLT is always around 8 records/second no matter how many worker I add to the DLT...
- 4064 Views
- 4 replies
- 7 kudos
- 7 kudos
Please consider the following:consider having driver 2 times bigger than worker,check is S3 in the same region, is communicating via the private gateway (local IPs),enable S3 transfer acceleration,in ingestion please user autoloader as described here...
- 7 kudos
- 2913 Views
- 5 replies
- 2 kudos
Resolved! How to access tables created in 2017
In 2017 while working on my Masters degree, I created some tables that I would like to access again. Back then I could just write SQL and find them but today that doesn't work. I suspect it has something to do with Delta Lake. What do I have to do to...
- 2913 Views
- 5 replies
- 2 kudos
- 1160 Views
- 0 replies
- 20 kudos
From databricks runtime 10.5 ARRAY_SIZE function was added.
From databricks runtime 10.5 ARRAY_SIZE function was added.
- 1160 Views
- 0 replies
- 20 kudos
- 1658 Views
- 1 replies
- 1 kudos
Resolved! Unable to start cluster on E2 Workspace
Hello Community,I'm trying to create and start my first cluster on my E2 Databricks Workspace on AWS; however, the cluster is created but after STARTING the cluster immediately the cluster status goes to TERMINATING. Logs provided by Databricks show ...
- 1658 Views
- 1 replies
- 1 kudos
- 1 kudos
Update:It was an error on my side with the KMS key.
- 1 kudos
- 1384 Views
- 1 replies
- 6 kudos
Databricks Office Hours Our next Office Hours session is scheduled for May 18th from 8:00 am - 9:00am PT. Do you have questions about how to set up or...
Databricks Office HoursOur next Office Hours session is scheduled for May 18th from 8:00 am - 9:00am PT.Do you have questions about how to set up or use Databricks? Do you want to learn more about the best practices for deploying your use case or tip...
- 1384 Views
- 1 replies
- 6 kudos
- 6 kudos
Just registered!
- 6 kudos
- 1298 Views
- 0 replies
- 20 kudos
From Databricks runtime 10.5 you can get metadata using the hidden _metadata column. Currently, the column contains input files information (file_path...
From Databricks runtime 10.5 you can get metadata using the hidden _metadata column. Currently, the column contains input files information (file_path, file_name, file_size and file_modification_time)
- 1298 Views
- 0 replies
- 20 kudos
- 3401 Views
- 5 replies
- 1 kudos
Resolved! Can ADLS be mounted in DBFS using only ADLS account key?
I realise this is not an optimal configuration but I'm trying to pull together a POC and I'm not at the point that I wish to ask the AAD admins to create an application for OAuth authentication.I have been able to use direct references to the ADLS co...
- 3401 Views
- 5 replies
- 1 kudos
- 1 kudos
Hey there @Ashley Betts​ Thank you for posting your question. And you found the solution.This is awesome!Would you be happy to mark the answer as best so that other members can find the solution more quickly?Cheers!
- 1 kudos
- 1179 Views
- 0 replies
- 24 kudos
The new Databricks jobs matrix is awesome! but looking at it can be addictiveÂ
The new Databricks jobs matrix is awesome!but looking at it can be addictive
- 1179 Views
- 0 replies
- 24 kudos
- 8891 Views
- 8 replies
- 4 kudos
Resolved! How do I read the contents of a hidden file in a Spark job?
I'm trying to read a file from a Google Cloud Storage bucket. The filename starts with a period, so Spark assumes the file is hidden and won't let me read it.My code is similar to this:from pyspark.sql import SparkSession spark = SparkSession.build...
- 8891 Views
- 8 replies
- 4 kudos
- 4 kudos
I don't think there is an easy way to do this. You will also break very basic functionality (like being able to read Delta tables) if you were able to get around these constraints. I suggest you employ a rename job and then read.
- 4 kudos
| User | Count |
|---|---|
| 1613 | |
| 771 | |
| 349 | |
| 286 | |
| 253 |
