- 23573 Views
- 17 replies
- 16 kudos
Resolved! Databricks slack channel
I would like to know if there is a Databricks slack channel? And if not, would anyone be interested in joining one?
- 23573 Views
- 17 replies
- 16 kudos
- 16 kudos
I joined with same email and it says try different emails. How do I join?
- 16 kudos
- 4853 Views
- 1 replies
- 2 kudos
Discrepancies between official spark 3.3.2 and what's provided in Databricks Runtime 12.2 LTS leads to NoSuchMethodError when creating ParquetToSparkSchemaConverter
I am trying to run my spark-dependent application on a Databricks cluster:spark-3.3.2on a cluster with Databricks Runtime 12.2 LTS (spark-3.3.2, scala-2.12)I end up with:NoSuchMethodError: org.apache.spark.sql.execution.datasources.parquet.ParquetToS...
- 4853 Views
- 1 replies
- 2 kudos
- 2 kudos
I ran into the same issueDid you happen to find any fix?
- 2 kudos
- 6889 Views
- 3 replies
- 1 kudos
Resolved! When creating a cluster, ERROR: "Missing required field 'spark_version'" even though it was specified
Every time I try to create a cluster over the API I get this. Even when I am literally using the example JSON from the API documentation:```{ "cluster_name": "single-node-cluster", "node_type_id": "i3.xlarge", "spark_version": "7.6.x-scala2...
- 6889 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi, Matt. Would be super useful if you could provide a more complete code example on how it worked.
- 1 kudos
- 3928 Views
- 2 replies
- 0 kudos
Init Script not 'Absolute'
Hello,I am trying to use an init script from my repo workspace. The cluster is a 'personal compute' cluster with single user access mode with the following runtime version: '13.1 (includes Apache Spark 3.4.0, Scala 2.12)' I have 'Files in Repos' set ...
- 3928 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi, could you please enable init script logging and provide the elaborated logs here? Looks like it is not able to get the package from the repo. https://docs.databricks.com/clusters/init-scripts.html#init-script-loggingAlso, please tag @Debayan Mukh...
- 0 kudos
- 5325 Views
- 6 replies
- 6 kudos
Accreditation badge for AZURE Databricks Platform Architect
Dear support,after 48h I didn't receive the Accreditation Badge for AZURE Databricks Platform Architect. The id is E-02KZJV. Could you check please? I completed the "Azure Databricks Platform Architect Accreditation" learing plan and passed with >80%...
- 5325 Views
- 6 replies
- 6 kudos
- 6 kudos
Hi @NIKOLAOS BERDOS I assume you will receive the ticket number since you have initiated the ticket. That means your ticket has been raised.Thanks and Regards
- 6 kudos
- 3032 Views
- 2 replies
- 1 kudos
Purge utility for databricks
I have one requirement where I need to delete older data from a folder, and folder data can be partitioned or without partition. What approach to follow to build this utility?
- 3032 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @Vidula Khanna @Kaniz Fatma It does not answer my real question. We mount adls path in databricks where we read and write the data. Suppose I have one partitioned (partitioned on audit) folder like below. I want to build an utility to delete the...
- 1 kudos
- 3841 Views
- 2 replies
- 2 kudos
Resolved! Delta table transaction log
What if i delete the delta table in the 10th transaction(Checkpoint) will i be able to recover the table through time travel
- 3841 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @Yatharth Malhotra Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best ans...
- 2 kudos
- 3630 Views
- 3 replies
- 0 kudos
- 3630 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @stephane cochet Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answe...
- 0 kudos
- 1839 Views
- 1 replies
- 1 kudos
Why doesn't my job-cluster scale down?
Hi,Running on Databricks-AWS, I have a job running on a cluster with 3 workers, 2-cores each (r6i.large), with autoscaling enabled.The Spark job has two stages: (1) highly parallelizable, cpu-intensive stage. This stage takes 15 minutes.(2) a non-par...
- 1839 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @Yoav Ben Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.
- 1 kudos
- 1705 Views
- 1 replies
- 0 kudos
Why is the current path different when executing a Notebook from Repos vs Workspace?
When executing the same Notebook from Repos and Workspace, the current path (`os.getcwd()`) is different:Repos=/Workspace/Repos/USERID/REPONAME.ide/pricingWorkspace=/home/spark-1636b562-0ce7-473b-bb17-56The issue is the Databricks VScode extension re...
- 1705 Views
- 1 replies
- 0 kudos
- 0 kudos
Found a work around:``` Pythonimport osnotebook_dir = os.path.dirname(dbutils.notebook.entry_point.getDbutils().notebook().getContext().notebookPath().get())notebook_dir = "/Workspace" + notebook_dircurrent_dir = os.getcwd()if current_dir != notebook...
- 0 kudos
- 3943 Views
- 4 replies
- 4 kudos
How to use Databricks Spark with Ibis
Ibis now includes pyspark backend. However using the databricks-connect doesn't seem to work with Ibis. Here's the sample code the throws an error.from databricks.connect import DatabricksSessionfrom databricks.sdk.core import Configconfig = Config(p...
- 3943 Views
- 4 replies
- 4 kudos
- 4 kudos
Hi Khanna, No solution provided yet. Can escalate with your technical team for a suitable solution please?Thanks,Calvin
- 4 kudos
- 3518 Views
- 4 replies
- 3 kudos
I could not access the deltatable from databricks. Error : org.apache.spark.sql.AnalysisException:
Hi all,I am not able to access the delta table from the datbase.While I try to read the table via spark.read.table command, I get the below error:org.apache.spark.sql.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.Run...
- 3518 Views
- 4 replies
- 3 kudos
- 3 kudos
Hi @Karthikeyan Palanisamy I hope you are doing great!Sorry for the inconvenience. Please specify that the issue is related to Databricks Community or Community -Edition or share the screenshot. Kindly, do let me know on a priority basis.Have a grea...
- 3 kudos
- 3356 Views
- 3 replies
- 4 kudos
Resolved! Checklist or Document for Migrating from Dev to Prod
Hello Everyone!So we have a dev environment in Databricks and want to migrate it to prod. I need it o go over every single table, schema, notebooks, and artifacts in the databricks and make sure nothing is hard-coded for example or that there is noth...
- 3356 Views
- 3 replies
- 4 kudos
- 4 kudos
Hi @Parsa Bahraminejad Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best an...
- 4 kudos
- 8057 Views
- 12 replies
- 13 kudos
Resolved! Connect Databricks with SSAS multidimensional Project
Hi,I just wanted to connect Databricks with SSAS multidimensional Project/want to use Databricks connection string and provider. Is that possible to do in SSAS multidimensional Project?Which Provider I need to use?Or Is there any other workaround to ...
- 8057 Views
- 12 replies
- 13 kudos
- 13 kudos
Hi @Mehala M We haven't heard from you since the last response from @Werner Stinckens , and I was checking back to see if her suggestions helped you.Or else, If you have any solution, please share it with the community, as it can be helpful to oth...
- 13 kudos
- 2388 Views
- 3 replies
- 3 kudos
maxFilesPerTrigger not working in bronze to silver layer
Hi,I am using Matillion architecture where autoloader picks files from AWS S3 and saves in delta lake. Next layer picks the changes from delta lake and does some processing. I am able to set batch size in autoloader and its working. But in bronze to...
- 2388 Views
- 3 replies
- 3 kudos
- 3 kudos
Hi @Sanjay Jain , Could you try using a fresh checkpoint location if not already tried? Also, could you please check the logs what is the size of the micro batch it is currently processing?
- 3 kudos
