- 1880 Views
- 2 replies
- 3 kudos
Azure Databricks SQL Warehouse connection issue
When trying to start SQL Warehouse from my Azure pay-as-you-go subscription, I'm getting error about not enough vCPUs provisioned. Documentation says to increase quota at Azure portal - but it requires knowing type of vCPUs to provision. What type of...
- 1880 Views
- 2 replies
- 3 kudos
- 3 kudos
Hi @Yuliya Quintela​,Just a friendly follow-up. Did Rostislaw's response help you to resolve your question? if it did, please mark it as best. Otherwise, please let us know if you still need help.
- 3 kudos
- 11266 Views
- 9 replies
- 2 kudos
SQLAlchemy ORM Connection String Error
We tried to insert records to Delta table using ORM. It looks like only SQLAlchemy has option to connect to Delta table.We tried the following codefrom sqlalchemy import Column, String, DateTime, Integer, create_engine engine = create_engine("data...
- 11266 Views
- 9 replies
- 2 kudos
- 2 kudos
Hi @Frank Zhang​ , Please disregard the driver comment. The Python SQL Connector requires no driver. Just a pip install and you are good to go. The links you provided don't actually show a working example of using SQL Alchemy's ORM to connect to Data...
- 2 kudos
- 1171 Views
- 2 replies
- 0 kudos
Where to report a bug with Databricks ?
I have in issue in Pyspark.Pandas to report. Is there a github or some forum where I can register my issue?Here's the issue
- 1171 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Krishna Zanwar​ Does @Debayan Mukherjee​ response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?We'd love to hear from you.Thanks!
- 0 kudos
- 3567 Views
- 1 replies
- 2 kudos
Resolved! Converting Dataframe into Nested xml
e.g.dataframe is having firstname,lastname,middlename,id,salaryI need to convert dataframe in xml file but in nested format.output as nested xml<Name> <firatname> <middlename> <lastname> </Name><id></id><salary></salary>Anyone has ides ho...
- 3567 Views
- 1 replies
- 2 kudos
- 2 kudos
databricks has a xml connector:https://docs.databricks.com/data/data-sources/xml.htmlBasically you just define a df with the correct structure and write it to xml.To create a nested df, here you can find some info.
- 2 kudos
- 6502 Views
- 8 replies
- 9 kudos
repos issue
Why repos works on one workspace and doesn't on another workspace? both have repos enabled.
- 6502 Views
- 8 replies
- 9 kudos
- 9 kudos
Do you see any errors or what is the issue that you are facing? Could you please describe more about this problem?
- 9 kudos
- 2116 Views
- 3 replies
- 6 kudos
Resolved! Streaming pipeline orchestration
For a batch job I can use ADF and Databricks notebook activity to create a pipeline.Similarly what Azure stack I should use to run Structured streaming Databricks notebook for a production ready pipeline.
- 2116 Views
- 3 replies
- 6 kudos
- 4129 Views
- 1 replies
- 2 kudos
Resolved! Serverless or Managed
We have about 12k write/s and 1.5TB/mo compressed S3 data. How can we choose between Serverless vs managed? And what will be good way to project the cost? In serverless, how the machine and hours scaled or scheduled based on the load? If there is a l...
- 4129 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi @Frank Zhang​ How can we choose between Serverless vs managed? And what will be good way to project the cost? -- Once you enable the serverless feature on your workspace, by default the new warehouse will be created with a serverless option. If yo...
- 2 kudos
- 2529 Views
- 2 replies
- 1 kudos
Getting spark/scala versioning issues while running the spark jobs through Jar
We tried moving our scala script from standalone cluster to databricks platform. Our script is compatible with following version:Spark: 2.4.8 Scala: 2.11.12The databricks cluster has spark/scala following with version:Spark: 3.2.1. Scala: 2.121: we ...
- 2529 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @Monika Samant​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Than...
- 1 kudos
- 1692 Views
- 1 replies
- 2 kudos
Resolved! Badge not received for Databricks Lakehouse Fundamentals Accreditation
Hello!I cleared the assessment for Databricks Lakehouse Fundamentals Accreditationbut not received a badge. Kindly assist me with this
- 1692 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi @Juan Afanador​ Thank you for reaching out! Please submit a ticket to our Training Team here: https://help.databricks.com/s/contact-us?ReqType=training and our team will get back to you shortly.
- 2 kudos
- 1281 Views
- 1 replies
- 1 kudos
Resolved! Lakehouse Fundamentals badge not received
Hi I have finished Lakehouse Fundamentals assessment, received my completion certificate but so far did not receive a badge for it. Would you be able to assist please?
- 1281 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @Maciej Oleksy​ Thank you for reaching out! Please submit a ticket to our Training Team here: https://help.databricks.com/s/contact-us?ReqType=training and our team will get back to you shortly.
- 1 kudos
- 2766 Views
- 3 replies
- 0 kudos
How to restart Databricks Cluster at specific time?
Command available for restart but not at specific time.databricks clusters restart --cluster-id <>
- 2766 Views
- 3 replies
- 0 kudos
- 0 kudos
@Trushna Khatri​ adding some more information to prabakar. can you please let me know what is actual need of starting cluster during specific time. usually if you criteria is to use for jobs go with job cluster. here cluster start when ever your job ...
- 0 kudos
- 6491 Views
- 3 replies
- 1 kudos
Metastore - Access validation failed
Hi,When i create a metastore in aws databricks, i always have this error in the picture bellow.Eventhought i follow this link https://docs.databricks.com/data-governance/unity-catalog/get-started.html#cloud-tenant-setup-aws
- 6491 Views
- 3 replies
- 1 kudos
- 1 kudos
@Anatole Cadet​ looks your IAM role is not properly configured, can you please check
- 1 kudos
- 14187 Views
- 8 replies
- 3 kudos
Where is the cluster logs of the Databricks Jobs stored.
I'm running a scheduled job on Job clusters. I didnt mention the log location for the cluster. Where can we get the stored logs location. Yes, I can see the logs in the runs, but i need the logs location.
- 14187 Views
- 8 replies
- 3 kudos
- 2694 Views
- 3 replies
- 0 kudos
How to pass admin role via Account Console SAML SSO
The Account Console SAML SSO docs mention that the admin role must be specified in the identity provider response. However it's not clear which attribute to use for passing role info via SAML.What SAML attribute should the role be assigned to? What d...
- 2694 Views
- 3 replies
- 0 kudos
- 0 kudos
Hey there @Jameel A.​ Hope you are well. Just wanted to see if you were able to find an answer to your question and would you like to mark an answer as best? It would be really helpful for the other members too. Else please let us know if you need mo...
- 0 kudos
- 2293 Views
- 5 replies
- 0 kudos
Streaming Queries Failing Frequently in DBR 10.4 LTS for the Last Week
DBR 10.4 LTS is failing frequently due to GC overhead once in half an hour.Can anyone from Databricks Team let me know if we have some existing tickets or bugs.Note : We used the same configuration and same DBR for almost last 3 months.When checking ...
- 2293 Views
- 5 replies
- 0 kudos
- 0 kudos
hi @Vidula Khanna​ have raised a support ticket to ADB from client side. We can close this however it seems like DBR Version 11.2 and above has some fixes for the RocksDB memory leak based on communication with Databricks developer team
- 0 kudos