- 6234 Views
- 2 replies
- 3 kudos
Resolved! Databricks Community Edition - Enable Databricks SQL
Hi,I am trying to enable the Databricks SQL environment from the Community Edition workspace (using left menu pane). However, the options I see in the dropdown menu are or Data Science & Eng and Machine Learning workspaces/environments. Is Databricks...
- 6234 Views
- 2 replies
- 3 kudos
- 3 kudos
+1 on what Andrew said. We are not currently planning on enabling Databricks SQL in Community. Please use a trial account on one of the supported clouds.
- 3 kudos
- 6331 Views
- 1 replies
- 1 kudos
rename a mount point folder
I am reading the data from a folder /mnt/lake/customer where mnt/lake is the mount path referring to ADLS Gen 2, Now I would like to rename a folder from /mnt/lake/customer to /mnt/lake/customeraddress without copying the data from one folder to ano...
- 6331 Views
- 1 replies
- 1 kudos
- 1 kudos
https://docs.databricks.com/data/databricks-file-system.html#local-file-api-limitations this might help @Simhadri Raju​
- 1 kudos
- 7174 Views
- 4 replies
- 4 kudos
- 7174 Views
- 4 replies
- 4 kudos
- 4 kudos
Hi @Kaniz Fatma​ , I got the solution based on the @Hubert Dudek​ Answer .Thanks @Hubert Dudek​ . Another Doubt:How do i Automate the Azure Synapse Concept . Please help me ..Thanks
- 4 kudos
- 8574 Views
- 4 replies
- 2 kudos
Resolved! Data persistence, Dataframe, and Delta
I am new to databricks platform. what is the best way to keep data persistent so that once I restart the cluster I don't need to run all the codes again?So that I can simply continue developing my notebook with the cached data.I have created many dat...
- 8574 Views
- 4 replies
- 2 kudos
- 2 kudos
Hey there @Vivek Ranjan​ Hope you are doing great!Just wanted to check in if you were able to resolve your issue or do you need more help? We'd love to hear from you.Thanks!
- 2 kudos
- 4418 Views
- 0 replies
- 0 kudos
Retrieve a row from indexed spark data frame.
Hello guys, I'm having an issue when trying to get a row values from spark data frame.I have a DF with index column, and i need to be able to return a row based on index in fastest way possible .I tried to partitionBy index column, optimize with zor...
- 4418 Views
- 0 replies
- 0 kudos
- 1746 Views
- 0 replies
- 0 kudos
Question on single job with multi task
Say, I have a job with 10 parallel tasks. I had to cancel one of the tasks to fix something and I unable to restart just that task. Is this by design? Should I restart the job in this case.Q2) If one of the tasks fails, will it auto recover just tha...
- 1746 Views
- 0 replies
- 0 kudos
- 8572 Views
- 2 replies
- 0 kudos
- 8572 Views
- 2 replies
- 0 kudos
- 0 kudos
@Amine El Helou​ when this limit will be lifted? my users won't switch from SSMS to dbx sql unless they can see all results set in UI ( which sometimes is more than 10mm rows)
- 0 kudos
- 10125 Views
- 6 replies
- 2 kudos
Resolved! Exploring additional cost saving options for structured streaming 24x7x365 uptime workloads
I currently have multiple jobs (each running its own job cluster) for my spark structured streaming pipelines that are long running 24x7x365 on DBR 9.x/10.x LTS. My SLAs are 24x7x365 with 1 minute latency. I have already accomplished the following co...
- 10125 Views
- 6 replies
- 2 kudos
- 2 kudos
Autoscaling doesn't work with structured streaming, so that's not really an option. Autoscaling is based on jobs sitting in the jobs queue for a long time, but that's not the case with streaming. Streaming is more many frequent small jobs. Spot in...
- 2 kudos
- 10504 Views
- 4 replies
- 2 kudos
Resolved! Databricks Spark XML parser : support for namespace declared at the ancestor level.
I'm trying to use Spark-XML API and I'm facing issue with the XSD validation option.Actually when I parser an XML file using the "rowValidationXSDPath" option the parser can't recognize the Prefixes/Namespaces declared at the root level. For this to...
- 10504 Views
- 4 replies
- 2 kudos
- 2 kudos
Hi sorry for the late response got busy looking for a permanent solution to this problem .At the end we are giving up on the XSDpath parser. This option does not work when Prefixes namespaces are declared at the ancestor level .Thank you anyway for ...
- 2 kudos
- 7604 Views
- 4 replies
- 6 kudos
Resolved! SELECT * FROM delta doesn't work on Spark 3.2
Using DBR 10 or later and I’m getting an error when running the following querySELECT * FROM delta.`s3://some_path`getting org.apache.spark.SparkException: Unable to fetch tables of db deltaFor 3.2.0+ they recommend reading like this:CREATE TEMPORAR...
- 7604 Views
- 4 replies
- 6 kudos
- 6 kudos
Got support from Databricks.Unfortunately, someone created a DB called delta, so the query was done against that DB instead. Issue was solved
- 6 kudos
- 10759 Views
- 9 replies
- 5 kudos
Overall security/access rights concept needed (combine Table Access Control and Credential Passthrough), how to allow users the benefits of both worlds
What we have:Databricks Workspace Premium on AzureADLS Gen2 storage for raw data, processed data (tables) and files like CSV, models, etc.What we want to do:We have users that want to work on Databricks to create and work with Python algorithms. We d...
- 10759 Views
- 9 replies
- 5 kudos
- 5 kudos
Hey @Vartika Nain​ , we are still at the same situation as described above. The Hive Metastore is a weak point.I would love to have the functionality that a mount can be dedicated to a given cluster.Regards, Gerhard
- 5 kudos
- 9314 Views
- 4 replies
- 4 kudos
Datepicker widget
There are textbox and dropdown list widgets in Databricks. Is there any datepicker widget? If not, is there any plan to add it?
- 9314 Views
- 4 replies
- 4 kudos
- 4 kudos
@Reza Rajabi​ , No, it is not I think at some meeting, someone discussed it. We can ask about it during the following office hours https://databricks.com/p/webinar/databricks-office-hours?utm_source=databricks&utm_medium=email&utm_campaign=7013f0000...
- 4 kudos
- 26868 Views
- 3 replies
- 2 kudos
Resolved! get a "Py4JJavaError: An error occurred while calling o5082.csv." when trying to save to csv file.
Hi, I'm trying to save a dataframe to csv with the code below:output.coalesce(1).write.mode('overwrite').option('header', 'true').csv(tmp_file_path) But it get "Py4JJavaError: An error occurred while calling o5082.csv." error. Any idea how to solve...
- 26868 Views
- 3 replies
- 2 kudos
- 17616 Views
- 4 replies
- 4 kudos
Resolved! Bucketing on Delta Tables
getting error as below while creating buckets on delta table.Error in SQL statement: AnalysisException: Delta bucketed tables are not supported.have fall back to parquet table due to this for some use cases. is their any alternative for this. i have...
- 17616 Views
- 4 replies
- 4 kudos
- 4 kudos
Hi @Rahul Samant​ , we checked internally on this due to certain limitations bucketing is not supported on delta tables, the only alternative for bucketing is to leverage the z ordering, below is the link for reference https://docs.databricks.com/de...
- 4 kudos
- 7423 Views
- 3 replies
- 2 kudos
Resolved! Spark Streaming - only process new files in streaming path?
In our streaming jobs, we currently run streaming (cloudFiles format) on a directory with sales transactions coming every 5 minutes.In this directory, the transactions are ordered in the following format:<streaming-checkpoint-root>/<transaction_date>...
- 7423 Views
- 3 replies
- 2 kudos
- 2 kudos
Update:Seems that maxFileAge was not a good idea. The following with the option "includeExistingFiles" = False solved my problem:streaming_df = ( spark.readStream.format("cloudFiles") .option("cloudFiles.format", extension) .option("...
- 2 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 584 | |
| 349 | |
| 287 |
