- 5491 Views
- 3 replies
- 0 kudos
Reading from /tmp no longer working
Since yesterday, reading a file copied into the cluster is no longer working.What used to work:blob = gcs_bucket.get_blob("dev/data.ndjson") -> worksblob.download_to_filename("/tmp/data-copy.ndjson") -> worksdf = spark.read.json("/tmp/data-copy.ndjso...
- 5491 Views
- 3 replies
- 0 kudos
- 0 kudos
I encountered this same issue, and figured out a fix!For some reason, it seems like only %sh cells can access the /tmp directory. So I just did...%sh ch /tmp/<file> /dbfs/<desired-location> and then accessed it form there using Spark.
- 0 kudos
- 2514 Views
- 3 replies
- 2 kudos
I haven't received certificate or badge after clearing data engineer associate exam
Hello Databricks Team,I completed my data engineer associate exam & passed the exam. But still now I haven't received certificate or badge. Also created help ticket, Ticket Number: #[00232273] Mail id : satishshravan888@gmail.comThanks,Satish
- 2514 Views
- 3 replies
- 2 kudos
- 2 kudos
Hi @Satish Shravan Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Tha...
- 2 kudos
- 17434 Views
- 3 replies
- 6 kudos
Resolved! "User not authorized" error when trying to deploy access connector on Azure
Hi,I have been trying to deploy Access Connector resource on Azure using Azure Pipelines (YAML) and a Bicep template but I cannot find a solution to this error:ERROR: {"status":"Failed","error":{"code":"DeploymentFailed","message":"At least one resou...
- 17434 Views
- 3 replies
- 6 kudos
- 6 kudos
Hi,I fixed this issue by adding the service principal to the list of service principals in the Account Console. My guess is that after the access connector is created an API call is made to the Databricks account and the service principal making that...
- 6 kudos
- 4656 Views
- 2 replies
- 2 kudos
implement autoloader to ingest data into delta lake, i have 100 different tables with full load , append merge senarios
i want to implement autoloader to ingest data into delta lake from 5 different source systems and i have 100 different tables in each database how do we dynamically address this by using autoloader , trigger once option - full load , append merge sen...
- 4656 Views
- 2 replies
- 2 kudos
- 2 kudos
You can create a generic notebook that will be parametrized with the table name/source system and then just simply trigger notebook with different parameters (for each table/source system).For parametrization you can use dbutils.widgets (https://docs...
- 2 kudos
- 6838 Views
- 2 replies
- 4 kudos
Why does Databricks SQL drop ending 0 in decimal data type
All,I have a column, RateAdj that is defined as DECIMAL(15,5) and I can see that the value is 4.00000, but when this gets inserted into my table it shows as just 4.%sql SELECT LTRIM(RTRIM(IFNULL(FORMAT_NUMBER(RateADJ, '0.00000'), '0.00000')))This i...
- 6838 Views
- 2 replies
- 4 kudos
- 4 kudos
Hi @Greg Bradley Hope everything is going great.Checking in. If @Uma Maheswara Rao Desula answer helped, would you let us know and mark the answer as best? If not, would you be happy to give us more information?Thanks!
- 4 kudos
- 3685 Views
- 1 replies
- 0 kudos
Resolved! Error Using spark.catalog.dropTempView()
I have a set of Spark Dataframes that I convert into Temp Views to run Spark SQL with. Then, I delete them after my logic/use is complete. The delete step throws an odd error that I am not sure how to fix. Looking for some tips on fixing it. As a not...
- 3685 Views
- 1 replies
- 0 kudos
- 0 kudos
spark.sql("DROP TABLE "+prefix_updates) spark.sql("DROP TABLE "+prefix_main)Fixed it for me.
- 0 kudos
- 5054 Views
- 4 replies
- 4 kudos
Passed Data Engineer Associate Exam but received twice the Lakehouse Accreditation
Hi all,today (27/12/22 14:00 Rome Time Zone) I passed the Data Engineer Associate exam, but I received the badge of Lakehouse Fundamentals (second time). My email address is: sisbarra@gmail.comMy company address is: antonio.sisbarra@nttdata.comCan ...
- 5054 Views
- 4 replies
- 4 kudos
- 4 kudos
Hello Antonio,I deleted the badge under: antonio.sisbarra@nttdata.com . You are good to go.Thanks!
- 4 kudos
- 9732 Views
- 9 replies
- 14 kudos
Resolved! Databricks Lakehouse Fundamentals Accreditation V2 badge/certificate not received.
Hello, this is regarding Databricks Lakehouse Fundamentals Accreditation V2. I haven't received my badge/certificate. I also raised a ticket but haven't received any response. My request id is #00248504.. Kindly help me out with this.
- 9732 Views
- 9 replies
- 14 kudos
- 14 kudos
@Jose Gonzalez ,@Vidula Khanna ,I also did not received my badge in portalThis is my portal link- https://credentials.databricks.com/profile/aviralbhardwaj143185/walletMy case number is - 00250939Also, my points are not updating in the reward port...
- 14 kudos
- 4022 Views
- 3 replies
- 2 kudos
How do we concatenate some fixed string with a secret value in Spark Config in Databricks Job Cluster?
Hi Team,I am trying to configure access to adls through Service Principal through Spark Config in Databricks job cluster. like, fs.azure.account.oauth2.client.id.<adls_account_name>.dfs.core.windows.net {{secrets/scopeName/clientID}}The above stateme...
- 4022 Views
- 3 replies
- 2 kudos
- 2 kudos
@Kaniz Fatma We are blocked on this issue. Can you please look into the thread and give your suggestion to workaround it.
- 2 kudos
- 31522 Views
- 1 replies
- 0 kudos
Resolved! How to show all rows by "DataFrame.show()"?
Hi,DataFrame.show() has a parameter n to set "Number of rows to show".Is there any way to show all rows?
- 31522 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi Medothis method will work fine df.show(df.count())
- 0 kudos
- 3267 Views
- 2 replies
- 3 kudos
Resolved! Can't do "Full screen" while taking Databricks Apache Spark developer course.
Hi, I see the option for "Full screen" on bottom right but its disabled/inactive. Attached is the screenshot for same.Please advise as its hard to read or see contents on half screen.Thanks
- 3267 Views
- 2 replies
- 3 kudos
- 3300 Views
- 3 replies
- 3 kudos
Access file location problem
import pandas as pd from apiclient.discovery import build from oauth2client.service_account import ServiceAccountCredentials df = spark.read.json("/FileStore/tables/cert.json") SCOPES = ['https://www.googleapis.com/auth/analytics.readonly'] KEY_FIL...
- 3300 Views
- 3 replies
- 3 kudos
- 3 kudos
Looks like it is because the oauth2client.service_account does not know about DBFS (whereas spark does).Is it an option to manage your secrets in databricks? https://docs.databricks.com/security/secrets/secrets.html
- 3 kudos
- 29797 Views
- 5 replies
- 7 kudos
Resolved! Best Practices for naming Tables and Databases in Databricks
We moved in Databricks since few months from now, and before that we were in SQL Server.So, all our tables and databases follow the "camel case" rule.Apparently, in Databricks the rule is "lower case with underscore".Where can we find an official doc...
- 29797 Views
- 5 replies
- 7 kudos
- 7 kudos
Hi @Salah KHALFALLAH , looking at the documentation it appears that Databricks' preferred naming convention is lowercase and underscores as you mentioned.The reason for this is most likely because Databricks uses Hive Metastore, which is case insens...
- 7 kudos
- 3171 Views
- 3 replies
- 3 kudos
Resolved! Why does chaining spark.read from one system/driver and .write to another system/driver take so much longer than doing each piece individually?
i am reading data from IBM DB2 and saving into a MS SQL server (the first step is moving the code itself to databricks, and then we will move the databases to databricks itself). Problem I'm running into is doing something like the below will take > ...
- 3171 Views
- 3 replies
- 3 kudos
- 3 kudos
Hi, it is related to partitioning optimization. By default, the JDBC driver queries the source database with only a single thread. So write was from one partition as one partition was created, so it was using a single core. When you used pandas, it d...
- 3 kudos
- 3164 Views
- 4 replies
- 4 kudos
RStudio on Databricks user experience
Is anybody actually using the RStudio app integration on Databricks? I'm surprised to find so little discussion in this forum. My team has been using it for about 3 months and it seems under-developed.1) No automated backup, you have to do it yoursel...
- 3164 Views
- 4 replies
- 4 kudos
- 4 kudos
@Jonathan Dufault Thanks for the response, and glad I'm not alone. My problem (and this is probably just a preference thing) is that the 'reward' of using a full-fledged IDE is huge, compared to bouncing between notebooks in multiple tabs. The integ...
- 4 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 555 | |
| 349 | |
| 287 |
