- 7382 Views
- 6 replies
- 0 kudos
Resolved! Databricks
Hi Ayapanday, How can check soure path in databricks
- 7382 Views
- 6 replies
- 0 kudos
- 0 kudos
Hi @Govardhana Reddy Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your feed...
- 0 kudos
- 6843 Views
- 6 replies
- 5 kudos
KeyProviderException when trying to create external table on databricks
Hi There,I have been trying to create an external table on Azure Databricks with below statement.df.write.partitionBy("year", "month", "day").format('org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat').option("path",sourcepath).mod...
- 6843 Views
- 6 replies
- 5 kudos
- 5 kudos
Hi @Gaurishankar Sakhare Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best ...
- 5 kudos
- 4152 Views
- 4 replies
- 4 kudos
- 4152 Views
- 4 replies
- 4 kudos
- 4 kudos
If you mean a stream-static join, yes that is possible:https://learn.microsoft.com/en-us/azure/databricks/delta-live-tables/transform#--stream-static-joinsIf not, what exactly do you mean?
- 4 kudos
- 1878 Views
- 1 replies
- 0 kudos
REPOS change my notebook format
Hi guys,I have some notebooks with REPOS but I noticed that REPOS changed my notebook format to .py because of this my Azure Data Factory no longer recognizes the notebook (.py)Have any ideia to convert that .py to databricks format ?
- 1878 Views
- 1 replies
- 0 kudos
- 0 kudos
that is odd. repos is merely another location (linked to git).You can copy/paste the code inside the py file into a notebook, or convert them using online tools or python libraries (like py2ipynb).
- 0 kudos
- 6291 Views
- 2 replies
- 3 kudos
Resolved! Unable to read data from ElasticSearch using Databricks (AWS) Cannot detect ES version - Caused by: org.elasticsearch.hadoop.rest.EsHadoopNoNodesLeftException: Connection error (check network and/or proxy settings)- all nodes failed; tried [IP:PORT]
I am trying to read data from ElasticSearch(ES Version 8.5.2) using PySpark on Databricks (13.0 (includes Apache Spark 3.4.0, Scala 2.12)). The ecosystem is on AWS.I am able to run a curl command on the Databricks notebook to the ES ip:port and fetch...
- 6291 Views
- 2 replies
- 3 kudos
- 1293 Views
- 0 replies
- 0 kudos
Dear Community- Get ready to mark your calendars for the upcoming Databricks Community Social event! Happening on June 16th, 2023, this event promis...
Dear Community- Get ready to mark your calendars for the upcoming Databricks Community Social event! Happening on June 16th, 2023, this event promises to be the ultimate monthly gathering for everyone in the Databricks Community.Join us for an hour ...
- 1293 Views
- 0 replies
- 0 kudos
- 876 Views
- 0 replies
- 0 kudos
Dear Community, Have you enrolled into the New Large Language Model Courses with edX yet?As Large Language Model (LLM) applications disrupt countless...
Dear Community, Have you enrolled into the New Large Language Model Courses with edX yet?As Large Language Model (LLM) applications disrupt countless industries, generative AI is becoming an important foundational technology. The demand for LLM-based...
- 876 Views
- 0 replies
- 0 kudos
- 1622 Views
- 1 replies
- 0 kudos
Databricks Certified Associate Developer for Apache Spark 3.0 - Python Cancellation
I have scheduled an examination on 1st June 2023 and due to personal reason, I have cancelled the examination on 26th May 2023 (more than 72 hours) but I am yet to receive the refund amount. In the auto generated mail it is mentioned that the refund ...
- 1622 Views
- 1 replies
- 0 kudos
- 0 kudos
adding @Suteja Kanuri and @Vidula Khanna for visibility
- 0 kudos
- 2642 Views
- 1 replies
- 0 kudos
Spark Dataframe write to Delta format doesn't create a _delta_log
Hello everyone, I have an intermittent issue when trying to create a Delta table for the first time in Databricks: all the data gets converted into parquet at the specified location but the _delta_log is not created or, if created, it's left empty, t...
- 2642 Views
- 1 replies
- 0 kudos
- 0 kudos
Can you list (display) the folder location "deltaLocation"? what files do you see here? have you try to use a new location for testing? do you get the same behavior?
- 0 kudos
- 2558 Views
- 1 replies
- 3 kudos
Can't access databricks
I was updated some scripts when I all of the sudden got a few "internal server errors". I refreshed the webpage a couple of times and now I am unable to login to databricks.When I try to sign in it thinks for a few seconds and then I am rerouted back...
- 2558 Views
- 1 replies
- 3 kudos
- 3 kudos
what date and time this issue happened? are you still unable to access to your workspace?
- 3 kudos
- 2277 Views
- 1 replies
- 2 kudos
Best sequence of using Vacuum, optimize, fsck repair and refresh commands.
I have a delta table whose size will increases gradually now we have around 1.5 crores of rows while running vacuum command on that table i am getting the below error.ERROR: Job aborted due to stage failure: Task 7 in stage 491.0 failed 4 times, most...
- 2277 Views
- 1 replies
- 2 kudos
- 2 kudos
Do you have access to the Executor 7 logs? is there a high GC or some other events that is making the heartbeat timeout? would you be able to check the failed stages?
- 2 kudos
- 11062 Views
- 1 replies
- 1 kudos
Resolved! Pydoc / Documentation Module
Does anyone have a recommendation for something along the lines of Pydoc that can be used to aggregate docstrings and the like into documentation pages?I tried Pydoc and it failed because of the magic commands in my repo.
- 11062 Views
- 1 replies
- 1 kudos
- 1 kudos
Sphinx could be an option here. It parses and render Databricks notebooks in the documentation. You might want to look into it to see if it fits your needs. However, it may not handle magic commands very well, and it assumes your notebooks are export...
- 1 kudos
- 2692 Views
- 2 replies
- 3 kudos
Terraform Global Init Script base64encoding
I am working on converting manual global init scripts into a terraform IaC process for multiple environments. Within terraform, we are using the resource "databricks_global_init_script" and set the content_base64 with the following:base64encoded(<<-...
- 2692 Views
- 2 replies
- 3 kudos
- 3 kudos
I am looking into it @Kristian Foster Are you able to get it working?
- 3 kudos
- 10180 Views
- 2 replies
- 3 kudos
Resolved! drop specific partition from a Delta Table
i have a delta table partitioned by a Date column , I'm trying to use the alter table drop partition command but get ALTER TABLE DROP PARTITION` is not supported for Delta tables erroris there a way to do it?
- 10180 Views
- 2 replies
- 3 kudos
- 3 kudos
Hi @Hanan Segal Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers y...
- 3 kudos
- 8096 Views
- 1 replies
- 2 kudos
Running into delta.exceptions.ConcurrentAppendException even after setting up S3 Multi-Cluster Writes environment via S3 Dynamo DB LogStore
My use-case is to process a dataset worth 100s of partitions in concurrency. The data is partitioned, and they are disjointed. I was facing ConcurrentAppendException due to S3 not supporting the “put-if-absent” consistency guarantee. From Delta Lake ...
- 8096 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi, You can refer to https://docs.databricks.com/optimizations/isolation-level.html#conflict-exceptions and recheck if everything is alright. Please let us know if this helps, also please tag @Debayan with your next response which will notify me, Th...
- 2 kudos
| User | Count |
|---|---|
| 1616 | |
| 786 | |
| 367 | |
| 349 | |
| 287 |
