Hello, i need to add a custom metadata into a avro file. The avro file containt data. we have tried to use "option" within the write function but it's not taken without generated any error.df.write.format("avro").option("avro.codec", "snappy").option...
I've been looking through the documentation and I swear this used to be listed somewhere, but for the life of me I can't find it anymore.
Thanks @Kaniz Fatma​
We have just started working with databricks in one of my university modules, and the lecturers gave us a set of commands to practice saving data in the FileStore. One of the commands was the following:dbutils .fs.cp("/ databricks - datasets / weathh...
you can delete files using the data explorer in Databricks web UI.another option is to use %fs or %sh in a notebook.
We have 11 people working on the Data Engineering Associate certification using Data Engineering with Databricks V3. We just got done with the Foundation one and start the Engineering journey. We are Registered partners and Data Engineering with Dat...
Hell Shanthala, you can send an email to partnerops@databricks.com who then provide information how to set this up
Hello,We have an Azure Data Factory pipeline running during the night, and one of the activities calls a Databricks Notebook with dynamic DatabricksInstancePoolId, ClusterVersion and Workers. Yesterday, it failed with with the following error:Cluster...
Hi @Rita Fernandes​,What are you trying to install in your init script? only the ODBC driver or some other libraries/dependencies?
POST to the slack webhook from local http client works_"Test" action on "System Notifcations" page in Databricks gives response 400 bad request
All in all, I got confused by a misconfiguration.A notification was setup as a webhook notificaton instead of a slack notification by mistake in a particular workspace, which caused the confusion. So, "problem in chair, not computer". If anything, it...
When trying to run a create table USING CSV that pulls data from an azure blob storage path in a custom catalog I created, I get an error stating Unsupported cloud file system schema 'wasbs'.However, when I run this code in the hive_metastore catalog...
Databricks Product Manager here  Would love to chat with anyone who has feedback on their experience onboarding to Databricks. Can't wait to hear from you!
How to list all the tables which has CDF enabled?I can review a table to find out if CDF is enable with below code.SHOW TBLPROPERTIES tableA(delta.enableChangeDataFeed)The return key ...
Hhi @Tim zhang​,Just a friendly follow-up. Did any of the responses help you to resolve your question? if it did, please mark it as best. Otherwise, please let us know if you still need help.
Recently to spark and databricks have been added support of star in groupBy.agg() statement. https://issues.apache.org/jira/browse/SPARK-41742
Hi,I use self managed VPC. when i use security group the jobs clusters are being created.if i make the default security group empty and create customized security group (required by soc2 to have empty default), with same definition of ingress (allow...
Hi @Amit Cahanovich​,Just a friendly follow-up. Did any of the responses help you to resolve your question? if it did, please mark it as best. Otherwise, please let us know if you still need help.
Hi, I need to remove the delete option from the data menu, how could I do it?
Hi @Andres Acuña​,Just a friendly follow-up. Did any of the responses help you to resolve your question? if it did, please mark it as best. Otherwise, please let us know if you still need help.
Please share link to take Lakehouse Fundamentals Accreditation (V2) in Databricks Academy.
i have mounted the adls container where the data are, but I cannot read the files with pandas ('pd.read_csv') even though I have attached the prefix '/dbfs/' to the path . Instead if I use 'spark.read.csv', I have no problems.Anyone knows why this i...
Hi, When you are not able to read the files, what is the error you get?
| User | Count |
|---|---|
| 1615 | |
| 779 | |
| 349 | |
| 287 | |
| 266 |
