cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Krish1
by New Contributor II
  • 1882 Views
  • 1 replies
  • 0 kudos

Loading multiple gz files from ADLS to Delta Lake/Delta table in ADB

I have several gz files (file.csv.gz) in a ADLS folder which are of pretty big size. All of these files are extractd from the same base table so it has the similar data but of different dates. How can I transfer them in delta lake/delta table. We wou...

  • 1882 Views
  • 1 replies
  • 0 kudos
Latest Reply
Debayan
Databricks Employee
  • 0 kudos

Hi, You can read GZ files through spark. https://stackoverflow.com/questions/42761912/how-to-read-gz-compressed-file-by-pysparkPlease let us know if this helps. Also, please tag @Debayan​ with your next comment so that I will get notified. Thank you!

  • 0 kudos
Dinu2
by New Contributor III
  • 2401 Views
  • 0 replies
  • 3 kudos

Cassandra connection from ADB

Hi , Could anyone please help to know the steps for connecting Cassandra from Azure Databricks? I have followed the steps in https://learn.microsoft.com/en-us/azure/databricks/_static/notebooks/azure/cassandra-azure.html But I am getting below error....

  • 2401 Views
  • 0 replies
  • 3 kudos
Soma
by Valued Contributor
  • 3740 Views
  • 3 replies
  • 3 kudos

Resolved! Unable to create Key Vault secrets scope with NPIP Workspace

Hi Team for secure connection we created secured cluster withNPIP(https://learn.microsoft.com/en-us/azure/databricks/security/secure-cluster-connectivity) WORKSPACE hosted in a private VNET.We had a hub vnet with private endpoint for key vault ,We pe...

  • 3740 Views
  • 3 replies
  • 3 kudos
Latest Reply
Debayan
Databricks Employee
  • 3 kudos

Hi @somanath Sankaran​ , did you face any error? if yes, could you please paste the error snapshot here?

  • 3 kudos
2 More Replies
Anonymous
by Not applicable
  • 3456 Views
  • 5 replies
  • 3 kudos

Encryption/Decryption options in ADB

Hello all,We are working on one of the client requirements to implement suitable data encryption in Azure Databricks.We should be able to encrypt and decrypt the data based on the access, we explored fernet library but client denied it saying it degr...

  • 3456 Views
  • 5 replies
  • 3 kudos
Latest Reply
Vidula
Honored Contributor
  • 3 kudos

Hi @purushotham Chanda​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you...

  • 3 kudos
4 More Replies
SailajaB
by Valued Contributor III
  • 1823 Views
  • 2 replies
  • 8 kudos

Resolved! How to restrict Azure users to use launch workspace to login to ADB workspace as admin when user has owner or contributor role

HI,Is there any way to disable launch workspace option in Azure portal for ADB.We have user accesses at resource group, so we need to restrict users who are part of owner or contributor role to launch ADB worksapce as admin.Thank you

  • 1823 Views
  • 2 replies
  • 8 kudos
Latest Reply
none_ranjeet
New Contributor III
  • 8 kudos

Deny Assignments don't block subscription contributor to launch workspace and become admin. Actually I haven't find any way to block that after many tries of different methods.

  • 8 kudos
1 More Replies
pawelmitrus
by Contributor
  • 1718 Views
  • 1 replies
  • 2 kudos

github.com

Do I always need to manually invite members of my AAD tenant to ADB workspace, if I don't have SCIM integration configured?EDIT: solved, it works when you go through Azure Portal and get in with "Launch Workspace" button on the ADB resource overview ...

  • 1718 Views
  • 1 replies
  • 2 kudos
Latest Reply
User16764241763
Honored Contributor
  • 2 kudos

Hello @pawelmitrus​ For users with Owner or Contributor roles, they should click on the "Launch Workspace" button in the Azure portal. For other users they should be explicitly granted access to the workspace to be able to login.Regards,Arvind

  • 2 kudos
thushar
by Contributor
  • 4106 Views
  • 8 replies
  • 6 kudos

Resolved! Compile all the scripts under the workspace folder

In workspace one folder I have around 100+ pyspark scripts, all these scripts need to be compiled before running the main program. In order to compile all these files, we are using the %run magic command like %run ../prod/netSales. Since we have 100+...

  • 4106 Views
  • 8 replies
  • 6 kudos
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 6 kudos

Problem is that you can list all files in workspace only via API call and than you can run every one of them using:dbutils.notebook.run()This is the script to list files from workspace (probably you need to add some filterning):import requests ctx = ...

  • 6 kudos
7 More Replies
Rithwik_Malla_2
by New Contributor
  • 2603 Views
  • 3 replies
  • 5 kudos

Resolved! Terraform - Databricks CI/CD pipeline

Can anyone help me in configuring the CI/CD for a ADB terraform code .The problem I am facing authentication something went wrong in there . Can anyone help me on this .Pipeline implementation in Azure DevOps.

  • 2603 Views
  • 3 replies
  • 5 kudos
Latest Reply
Ravi
Databricks Employee
  • 5 kudos

Hi @Rithwik Aditya Manoj Malla​ , as requested by @Prabakar Ammeappin​ earlier could you please share the code block and the error details. Also, you can refer to the below doc to authenticate ADB from TF code.https://registry.terraform.io/providers/...

  • 5 kudos
2 More Replies
Labels