I'm attempting to migrate data from Azure Data Lake to S3 using deep clone. The data in the source Data Lake is stored in Parquet format and partitioned. I've tried to follow the documentation from Databricks, which suggests that I need to register ...
Hi,Our scheduled job runs into below error once in a while and job fails. Any leads or thoughts please why we run into this once in a while and how to fix it pleaseshaded.databricks.org.apache.hadoop.fs.s3a.DatabricksThrottledException: Instantiate s...
Hello team, I am not able to launch databricks cluster in community edition. automatically its getting terminated. Can someone please help here ? Regards.,poonam
I face the exact same problem. The message i get is this:"Bootstrap Timeout:Node daemon ping timeout in 780000 ms for instance i-062042a9d4be8725e @ 10.172.197.194. Please check network connectivity between the data plane and the control plane."
Hi databricks:I have a job where I am saving my data in json format lzo compressed which requires the library lzo-codecon shifting to graviton instances I noticed that the same job started throwing exceptionCaused by: java.lang.RuntimeException: nati...
For more context, Please use the following code to replicate the error:# Create a Python list containing JSON objectsjson_data = [ { "id": 1, "name": "John", "age": 25 }, { "id": 2, "name": "Jane", "...
When saving notebook to GiHub repo, it is stripped to Python source code. Is it possible to save it in the ipynb formt?
When I save+commit+push my .ipynb file to my linked git repo, I noticed that only the cell inputs are saved, not the output. This differs from the .ipynb file I get when I choose "File / Export / iPython Notebook". Is there a way to save the cell o...
When I save+commit+push my .ipynb file to my linked git repo, I noticed that only the cell inputs are saved, not the output. This differs from the .ipynb file I get when I choose "File / Export / iPython Notebook". Is there a way to save the cell o...
I may have figured this out. You need to allow output in the settings, which will add a .databricks file to your repo, then you'll need to edit the options on your notebook and/or edit the .databricks file to allow all outputs.
Hello,I'm encountering an issue while running a notebook that utilizes the Pulp library. The library is installed in the first cell of the notebook. Occasionally, I encounter the following error: org.apache.spark.SparkException: Job aborted due to s...
I just tried to setup a row filter via the following two sql snippets:create function if not exists foo.my_test.row_filter (batch_id bigint) return TRUE;alter table foo.my_test.some_table set row filter foo.my_test.row_filter on (batch_id); This resu...
To be fair, row filters and the check constraints feature are in Public Preview, so I apologize for the slightly harsh words above!
Hello community,Using Databricks SQL for the first time and I was hoping I could just copy and past my queries from SSMS across and update the table names, but it's not working.Found it's the where statement, which I updated the ' ' to " " but still ...
Hi, were you able to get this figured out? I am having the same issue.
I already reported that as a Bug to the official Spark bug tracker: https://issues.apache.org/jira/browse/SPARK-46876A short summary: When reading a tab separated file, that has lines that only contain of tabs, then this line will not show up in the ...
@Lakshay Do you know any way to speed up the github merge/review process? The issue has a proposed fix since more than 4 weeks now, but no one seems to care...
Hi! Today working trying to collect some metrics to create a splot in my spark structure streaming. It is configured with a trigger(processingTime="30 seconds") and I am trying to collect data with the following Listener Class (just an example). # D...
Hi,I have done further investigation on this.Below I have tried to illustrate the issue through PySpark code def onQueryProgress(self, event): print("onQueryProgress") # Access micro-batch data microbatch_data = event.progre...
Hi, We have created a Databricks Jobs in Workflows where concurrent runs are set to 10 and the queue is enabled. We were trying to perform concurrent users testing by triggering 100 job runs using Jmeter script. We have observed that the first 10 job...
I am ingesting json files from S3 using Autoloader and would like to use schemaHints to define the datatype of one of the fields, that is, I wanted the field id to be of integer type.The DLT code below infers the the id as string, with correct values...
I have a scenario to implement using the delta live tables.I get the id and timestamp column from source and I have to load that into my delta live streaming output table only if the source timestamp for less that the existing value in the output tab...
What are the best practices for using MLOps and DLT together?This page https://learn.microsoft.com/en-us/azure/databricks/compute/access-mode-limitations states that You cannot use a single user cluster to query tables created by a Unity Catalog-enab...
| User | Count |
|---|---|
| 1611 | |
| 768 | |
| 348 | |
| 286 | |
| 252 |
