cancel
Showing results for 
Search instead for 
Did you mean: 
missing-QuestionPost
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

396827
by New Contributor II
  • 724 Views
  • 1 replies
  • 2 kudos

Discrepancies between official spark 3.3.2 and what's provided in Databricks Runtime 12.2 LTS leads to NoSuchMethodError when creating ParquetToSparkSchemaConverter

I am trying to run my spark-dependent application on a Databricks cluster:spark-3.3.2on a cluster with Databricks Runtime 12.2 LTS (spark-3.3.2, scala-2.12)I end up with:NoSuchMethodError: org.apache.spark.sql.execution.datasources.parquet.ParquetToS...

  • 724 Views
  • 1 replies
  • 2 kudos
Latest Reply
OsamaNabih
New Contributor II
  • 2 kudos

I ran into the same issueDid you happen to find any fix?

  • 2 kudos
141402
by New Contributor II
  • 468 Views
  • 3 replies
  • 1 kudos

Have "viewer's credential" as default setting in Query access control, instead of "owner's credential"

Hi everybody,I am curious if there is a way to raise the security level by default. Specifically if it is possible to configure the default setting to limit the access of the query, rather than granting it open, as right now from my side the default ...

  • 468 Views
  • 3 replies
  • 1 kudos
Latest Reply
Kaniz
Community Manager
  • 1 kudos

Hi @Ralph_RevoData , Certainly! Ensuring better security by default is crucial, especially to prevent inadvertent information leaks.    Let’s explore how you can configure query access control in Databricks to limit query access permissions:   Query ...

  • 1 kudos
2 More Replies
Oliver_Angelil
by Valued Contributor
  • 4338 Views
  • 18 replies
  • 21 kudos

Resolved! Databricks slack channel

I would like to know if there is a Databricks slack channel? And if not, would anyone be interested in joining one?

  • 4338 Views
  • 18 replies
  • 21 kudos
Latest Reply
Oliver_Angelil
Valued Contributor
  • 21 kudos

UPDATE:Changed workspace name to databricks-users. Click here to join https://databricks-users.slack.comOr click this link (expires in 1 month) 

  • 21 kudos
17 More Replies
MattJohn_35453
by New Contributor III
  • 1403 Views
  • 3 replies
  • 0 kudos

Resolved! When creating a cluster, ERROR: "Missing required field 'spark_version'" even though it was specified

Every time I try to create a cluster over the API I get this. Even when I am literally using the example JSON from the API documentation:```{ "cluster_name": "single-node-cluster", "node_type_id": "i3.xlarge", "spark_version": "7.6.x-scala2...

  • 1403 Views
  • 3 replies
  • 0 kudos
Latest Reply
Norway
New Contributor II
  • 0 kudos

Hi, Matt. Would be super useful if you could provide a more complete code example on how it worked.

  • 0 kudos
2 More Replies
AleksandraFrolo
by New Contributor III
  • 230 Views
  • 0 replies
  • 0 kudos

Web scraping with Databricks

Hello,What is the easiest way to do web scraping in Databricks? Let's imagine that from this link: http://automated.pythonanywhere.com , I need to grab this element "/html/body/div[1]/div/h1[1]" and return a text, how can I do it? Can somebody write ...

  • 230 Views
  • 0 replies
  • 0 kudos
alexiswl
by Contributor
  • 1613 Views
  • 2 replies
  • 0 kudos

Init Script not 'Absolute'

Hello,I am trying to use an init script from my repo workspace. The cluster is a 'personal compute' cluster with single user access mode with the following runtime version: '13.1 (includes Apache Spark 3.4.0, Scala 2.12)' I have 'Files in Repos' set ...

  • 1613 Views
  • 2 replies
  • 0 kudos
Latest Reply
Debayan
Esteemed Contributor III
  • 0 kudos

Hi, could you please enable init script logging and provide the elaborated logs here? Looks like it is not able to get the package from the repo. https://docs.databricks.com/clusters/init-scripts.html#init-script-loggingAlso, please tag @Debayan Mukh...

  • 0 kudos
1 More Replies
berdoni
by New Contributor III
  • 1901 Views
  • 6 replies
  • 6 kudos

Accreditation badge for AZURE Databricks Platform Architect

Dear support,after 48h I didn't receive the Accreditation Badge for AZURE Databricks Platform Architect. The id is E-02KZJV. Could you check please? I completed the "Azure Databricks Platform Architect Accreditation" learing plan and passed with >80%...

  • 1901 Views
  • 6 replies
  • 6 kudos
Latest Reply
Anonymous
Not applicable
  • 6 kudos

Hi @NIKOLAOS BERDOS​ I assume you will receive the ticket number since you have initiated the ticket. That means your ticket has been raised.Thanks and Regards

  • 6 kudos
5 More Replies
vijaykumar99535
by New Contributor III
  • 474 Views
  • 3 replies
  • 1 kudos

Purge utility for databricks

I have one requirement where I need to delete older data from a folder, and folder data can be partitioned or without partition. What approach to follow to build this utility?

  • 474 Views
  • 3 replies
  • 1 kudos
Latest Reply
vijaykumar99535
New Contributor III
  • 1 kudos

Hi @Vidula Khanna​ @Kaniz Fatma​ It does not answer my real question. We mount adls path in databricks where we read and write the data. Suppose I have one partitioned (partitioned on audit) folder like below. I want to build an utility to delete the...

  • 1 kudos
2 More Replies
yatharth
by New Contributor III
  • 644 Views
  • 2 replies
  • 2 kudos

Resolved! Delta table transaction log

What if i delete the delta table in the 10th transaction(Checkpoint) will i be able to recover the table through time travel

  • 644 Views
  • 2 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @Yatharth Malhotra​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best ans...

  • 2 kudos
1 More Replies
chico
by New Contributor II
  • 1055 Views
  • 3 replies
  • 0 kudos
  • 1055 Views
  • 3 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @stephane cochet​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answe...

  • 0 kudos
2 More Replies
804925
by New Contributor II
  • 368 Views
  • 1 replies
  • 1 kudos

Why doesn't my job-cluster scale down?

Hi,Running on Databricks-AWS, I have a job running on a cluster with 3 workers, 2-cores each (r6i.large), with autoscaling enabled.The Spark job has two stages: (1) highly parallelizable, cpu-intensive stage. This stage takes 15 minutes.(2) a non-par...

  • 368 Views
  • 1 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Yoav Ben​ Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.

  • 1 kudos
Duy
by New Contributor II
  • 439 Views
  • 1 replies
  • 0 kudos

Why is the current path different when executing a Notebook from Repos vs Workspace?

When executing the same Notebook from Repos and Workspace, the current path (`os.getcwd()`) is different:Repos=/Workspace/Repos/USERID/REPONAME.ide/pricingWorkspace=/home/spark-1636b562-0ce7-473b-bb17-56The issue is the Databricks VScode extension re...

  • 439 Views
  • 1 replies
  • 0 kudos
Latest Reply
Duy
New Contributor II
  • 0 kudos

Found a work around:``` Pythonimport osnotebook_dir = os.path.dirname(dbutils.notebook.entry_point.getDbutils().notebook().getContext().notebookPath().get())notebook_dir = "/Workspace" + notebook_dircurrent_dir = os.getcwd()if current_dir != notebook...

  • 0 kudos
ccbeloy
by New Contributor III
  • 1105 Views
  • 4 replies
  • 3 kudos

How to use Databricks Spark with Ibis

Ibis now includes pyspark backend. However using the databricks-connect doesn't seem to work with Ibis. Here's the sample code the throws an error.from databricks.connect import DatabricksSessionfrom databricks.sdk.core import Configconfig = Config(p...

  • 1105 Views
  • 4 replies
  • 3 kudos
Latest Reply
ccbeloy
New Contributor III
  • 3 kudos

Hi Khanna, No solution provided yet. Can escalate with your technical team for a suitable solution please?Thanks,Calvin

  • 3 kudos
3 More Replies
Karthe
by New Contributor III
  • 1115 Views
  • 5 replies
  • 4 kudos

I could not access the deltatable from databricks. Error : org.apache.spark.sql.AnalysisException:

Hi all,​I am not able to access the delta table from the datbase.While I try to read the table via spark.read.table command, I get the below error:org.apache.spark.sql.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.Run...

  • 1115 Views
  • 5 replies
  • 4 kudos
Latest Reply
Anonymous
Not applicable
  • 4 kudos

Hi @Karthikeyan Palanisamy​ I hope you are doing great!Sorry for the inconvenience. Please specify that the issue is related to Databricks Community or Community -Edition or share the screenshot. Kindly, do let me know on a priority basis.Have a grea...

  • 4 kudos
4 More Replies
Enzo_Bahrami
by New Contributor III
  • 649 Views
  • 3 replies
  • 4 kudos

Resolved! Checklist or Document for Migrating from Dev to Prod

Hello Everyone!So we have a dev environment in Databricks and want to migrate it to prod. I need it o go over every single table, schema, notebooks, and artifacts in the databricks and make sure nothing is hard-coded for example or that there is noth...

  • 649 Views
  • 3 replies
  • 4 kudos
Latest Reply
Anonymous
Not applicable
  • 4 kudos

Hi @Parsa Bahraminejad​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best an...

  • 4 kudos
2 More Replies