cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Jennifer
by New Contributor III
  • 4254 Views
  • 5 replies
  • 0 kudos

Resolved! Import python file to notebook doesn't work

I followed the documentation here under the section "Import a file into a notebook" to import a shared python file among notebooks used by delta live table. But it sometimes can find the module, sometimes not and returns me exception No module named ...

  • 4254 Views
  • 5 replies
  • 0 kudos
Latest Reply
Vartika
Moderator
  • 0 kudos

Thank you so much for getting back to us @Jennifer MJ​ . It's really great of you to send in the solution. Would you be happy to mark the answer as best so other community members can find the solution quickly and easily? We really appreciate your ti...

  • 0 kudos
4 More Replies
Kayla
by Contributor
  • 1574 Views
  • 1 replies
  • 1 kudos

Resolved! Pydoc / Documentation Module

Does anyone have a recommendation for something along the lines of Pydoc that can be used to aggregate docstrings and the like into documentation pages?I tried Pydoc and it failed because of the magic commands in my repo.

  • 1574 Views
  • 1 replies
  • 1 kudos
Latest Reply
User16539034020
Contributor II
  • 1 kudos

Sphinx could be an option here. It parses and render Databricks notebooks in the documentation. You might want to look into it to see if it fits your needs. However, it may not handle magic commands very well, and it assumes your notebooks are export...

  • 1 kudos
matt_at_ark
by New Contributor
  • 1081 Views
  • 0 replies
  • 0 kudos

Bokeh plotting with ipywidgets or in Databricks Dashboard

Databricks' documentation for using bokeh shows using displayHTML, which seems to be incompatible with using ipywidgets, or putting them in Databricks dashboards.I found there is undocumented functionality with plotting interactive altair charts in D...

  • 1081 Views
  • 0 replies
  • 0 kudos
DavideCagnoni
by Contributor
  • 4136 Views
  • 1 replies
  • 4 kudos

Resolved! How to use multi-cursor and rectangular selection for notebooks and query editor in Linux ?

The documentation explains how to use multicursor in notebooks. However, it only says it for Windows and MacOS. The Windows way would work in Linux (Ubuntu) up to a few days ago but it does not work now anymore.

  • 4136 Views
  • 1 replies
  • 4 kudos
Latest Reply
Anonymous
Not applicable
  • 4 kudos

@Davide Cagnoni​ :Multicursor support in Databricks notebooks is implemented using the Ace editor, which is a web-based code editor. Therefore, the behavior of multicursor support may depend on the specific browser and operating system you are using....

  • 4 kudos
Sulfikkar
by Contributor
  • 3234 Views
  • 8 replies
  • 4 kudos

Cluster Scoped init script through pulumi

I am trying to run a cluster-scoped init script through Pulumi. I have referred to this documentation https://learn.microsoft.com/en-us/azure/databricks/clusters/configure#spark-configuration However, looks like the documentation is not very clear.I ...

  • 3234 Views
  • 8 replies
  • 4 kudos
Latest Reply
Vivian_Wilfred
Honored Contributor
  • 4 kudos

Hi @Sulfikkar Basheer Shylaja​ , Why don't you store the init-script on DBFS and just pass the dbfs:/ path of the init script in Pulumi? You could just run this code on a notebook-%python dbutils.fs.put("/databricks/init-scripts/set-private-pip-repos...

  • 4 kudos
7 More Replies
asethia
by New Contributor
  • 2374 Views
  • 1 replies
  • 0 kudos

delta lake in Apache Spark

Hi,As per documentation https://docs.delta.io/latest/quick-start.html , we can configure DeltaCatalog using spark.sql.catalog.spark_catalog.The Iceberg supports two Catalog implementations (https://iceberg.apache.org/docs/latest/spark-configuration/#...

  • 2374 Views
  • 1 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

@Arun Sethia​ :Yes, Delta Lake also supports custom catalogs. Delta Lake uses the Spark Catalog API, which allows for pluggable catalog implementations. You can implement your own custom catalog to use with Delta Lake.To use a custom catalog, you can...

  • 0 kudos
JordanYaker
by Contributor
  • 1106 Views
  • 3 replies
  • 3 kudos

Resolved! What is the maximum number of workspaces per account using Databricks on AWS?

I've been looking through the documentation and I swear this used to be listed somewhere, but for the life of me I can't find it anymore.

  • 1106 Views
  • 3 replies
  • 3 kudos
Latest Reply
JordanYaker
Contributor
  • 3 kudos

Thanks @Kaniz Fatma​ 

  • 3 kudos
2 More Replies
chanansh
by Contributor
  • 883 Views
  • 1 replies
  • 0 kudos

QueryExecutionListener cannot be found in pyspark

According to the documentation you can monitor a spark structure stream job using QueryExecutionListener. However I cannot find it. https://docs.databricks.com/structured-streaming/stream-monitoring.html#language-python

  • 883 Views
  • 1 replies
  • 0 kudos
Latest Reply
jose_gonzalez
Moderator
  • 0 kudos

Which DBR version are you using? also, can you share some code snippet on how you are using the QueryExecutionListener?

  • 0 kudos
chanansh
by Contributor
  • 995 Views
  • 1 replies
  • 0 kudos

Resolved! autoloader documentation does not work

I am trying to following the documentation here:https://learn.microsoft.com/en-us/azure/databricks/getting-started/etl-quick-startMy code looks like:(spark.readStream .format("cloudFiles") .option("header", "true") #.option("cloudFiles.partitio...

  • 995 Views
  • 1 replies
  • 0 kudos
Latest Reply
Murthy1
Contributor II
  • 0 kudos

Hi,It seems like you are writing to a path which is not empty and has some non - delta format files. Also, can you confirm if the path mentioned in the error message "`s3://nbu-ml/projects/rca/msft/dsm09collectx/delta` " is the path you are writing t...

  • 0 kudos
Sujitha
by Community Manager
  • 501 Views
  • 1 replies
  • 2 kudos

Documentation Update January 13 - 19 Databricks documentation provides how-to guidance and reference information for data analysts, data scientists, a...

Documentation Update January 13 - 19Databricks documentation provides how-to guidance and reference information for data analysts, data scientists, and data engineers working in the Databricks Data Science & Engineering, Databricks Machine Learning, ...

  • 501 Views
  • 1 replies
  • 2 kudos
Latest Reply
Aviral-Bhardwaj
Esteemed Contributor III
  • 2 kudos

thanks for the details

  • 2 kudos
cybersam
by New Contributor II
  • 715 Views
  • 2 replies
  • 0 kudos

How do I find the documentation for a Databricks "platform release"?

My DB portal says the platform version is v3.86, and provides a link to all the releases. But none of those releases state the "platform version". And I can't find "v3.86" by searching in the Databricks docs.So, how does one find the documentation fo...

  • 715 Views
  • 2 replies
  • 0 kudos
Latest Reply
Aviral-Bhardwaj
Esteemed Contributor III
  • 0 kudos

Hey @Samuel Yang​ ,Here you will find all detailshttps://docs.databricks.com/release-notes/index.htmlThanksAviral Bhardwaj

  • 0 kudos
1 More Replies
Sujitha
by Community Manager
  • 760 Views
  • 1 replies
  • 4 kudos

Documentation Update  Databricks documentation provides how-to guidance and reference information for data analysts, data scientists, and data enginee...

Documentation Update Databricks documentation provides how-to guidance and reference information for data analysts, data scientists, and data engineers working in the Databricks Data Science & Engineering, Databricks Machine Learning, and Databricks ...

  • 760 Views
  • 1 replies
  • 4 kudos
Latest Reply
Harun
Honored Contributor
  • 4 kudos

Thanks for sharing @Sujitha Ramamoorthy​ 

  • 4 kudos
Gustavo_Az
by Contributor
  • 2808 Views
  • 9 replies
  • 27 kudos

Resolved! When will be available the course "Data Engineering with Databricks V3" in Databricks Academy?

In the documentation of the V2 it says that it would be released 4 days ago. But searching for it in the academy only throws results for V1 and V2.

  • 2808 Views
  • 9 replies
  • 27 kudos
Latest Reply
Anonymous
Not applicable
  • 27 kudos

@Gustavo Amadoz Navarro​  Updated new infoThis course will be part of the data engineer learning path once the Databricks Certified Data Engineer Associate V3 exam is released (November 19, 2022). BEFORE YOU GET STARTED: Please note that this course,...

  • 27 kudos
8 More Replies
Oliver_Floyd
by Contributor
  • 1313 Views
  • 4 replies
  • 6 kudos

Where to find documentation about : spark.databricks.driver.strace.enabled

Hello ,For a support request, Microsoft support ask me to add spark.databricks.driver.strace.enabled trueto my cluster configuration.MS was not able to send me a link to the documentation and I did not find it on the databricks website.Can someone he...

  • 1313 Views
  • 4 replies
  • 6 kudos
Latest Reply
Oliver_Floyd
Contributor
  • 6 kudos

Yes no problem. I have a python program, called "post ingestion", that run on a databricks job cluster during the night and consist of :inserting data to a deltalake tableexecuting an optimize command on that tableexecuting a vacuum command on that t...

  • 6 kudos
3 More Replies
db-avengers2rul
by Contributor II
  • 1120 Views
  • 3 replies
  • 2 kudos

Resolved! Documentation - notebook not working

Dear Team,While practising few examples i have noticed the below notebook is not fetching the full dataset and also no schema is fetched https://docs.databricks.com/_static/notebooks/widget-demo.htmlcan you please re try and let me know the results N...

  • 1120 Views
  • 3 replies
  • 2 kudos
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 2 kudos

I think https://health.data.ny.gov/api/views/myeu-hzra/rows.csv was a public dataset, but now it shows authentication_required, so this error is independent of databricks.But the good news is that I was able to generate a new URL on the New Your heal...

  • 2 kudos
2 More Replies
Labels